From gary.poster at canonical.com Sat May 1 00:01:32 2010 From: gary.poster at canonical.com (Gary Poster) Date: Fri, 30 Apr 2010 18:01:32 -0400 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: <4D19E04C-A053-497E-9473-459EB1B65D9C@canonical.com> I have re-released zc.buildout 1.4.3 as 1.5.0b2, and re-released zc.recipe.egg 1.2.2 as 1.2.3b2. My tests seem to show everything back to normal--I was able to build Chris's initial problem example as a smoketest. Next week I'll announce betas *not* uploaded to PyPI. These will solve the problems identified today. I'll provide instructions on how to download and install them locally for tests. I appreciate your feedback today, and will appreciate it even more when you try out those betas next week! In closing, I'll summarize the problems identified today with 1.5.0b1, with their statuses. - virtualenv + zc.buildout fails. 1.5.0b1 expected the -S of executables to work properly, and it doesn't for virtualenv. This may not be fixable within virtualenv, per Carl's email. I have a change to zc.buildout that I think will work around this in a local branch. Tests fail, so it needs more work. This is the most critical problem identified. - zc.buildout's bootstrap honors more options in the .cfg file, which caused Jonathan some grief. I'm classifying this as a "Won't Fix" unless I get pushback, per my earlier mail. - If you have buildout installed in your system Python's site-packages, bootstrap falls over. I have not duped this but I think I know the cause. I'll try to dupe, and ask Rodrigo to confirm a fix when I'm ready next week. - zc.recipe.testing's tests showed some API calls to zc.buildout that failed mysteriously. I have not investigated that yet. I know of several other existing recipes that use zc.buildout's API that work fine. I'm hopeful that this will be easy to diagnose and address. - Tarek raised the problem that we can't easily handle beta packages of critical packaging infrastructure. Jim countered with information on how buildout already does handle it. However, I don't think that this helps with bootstrapping. Moreover, it requires explicit configuration to get some of the behavior Jim describes. I plan to address Tarek's concerns for now by having betas available through alternate means than PyPI. However, this will require more active participation from you all to help prepare a smooth release on PyPI. Thank you all for your help. Talk to you next week. Gary From dave at krondo.com Sat May 1 05:55:54 2010 From: dave at krondo.com (Dave Peticolas) Date: Fri, 30 Apr 2010 20:55:54 -0700 Subject: [Distutils] zc.buildout 1.5.0b1 and zc.recipe.egg.1.2.3b1 Message-ID: <4BDBA64A.8080300@krondo.com> Thanks for the quick resolution of this, I just wanted to mention a problem I ran into with these that I don't think I saw reported in the archives. After working around the 'cannot import warnings' problem, I was getting errors where buildout complained about a missing find-links parameter. I set that to the empty value: [buildout] find-links = And then got the same thing with another one, can't remember which. I set that to the empty value, and then ran into a traceback about a parameter, I think it was 'include_site_packages' being passed to the working_set function that it didn't support. Sorry I can't be more specific about that, I was rebuilding my virtualenv several times, when it started working again thanks to the b2 releases. Since this was around the time the b2 versions were released, maybe it was a version mismatch between a b1 and b2 version? thanks, dave From ziade.tarek at gmail.com Sat May 1 10:16:16 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 1 May 2010 10:16:16 +0200 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: On Fri, Apr 30, 2010 at 11:02 PM, Gary Poster wrote: [..] >> >> I propose that we "hide" the beta version on PyPI until all problems >> are cleared u, so people don't auto-upgrade their buildouts. > > I have done so, for zc.buildout and zc.recipe.egg. ?However, merely hiding them is apparently insufficient, in my tests. Ooops, I thought hidden release weren't showing up in the index From ziade.tarek at gmail.com Sat May 1 10:31:01 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 1 May 2010 10:31:01 +0200 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: <4D19E04C-A053-497E-9473-459EB1B65D9C@canonical.com> References: <4D19E04C-A053-497E-9473-459EB1B65D9C@canonical.com> Message-ID: On Sat, May 1, 2010 at 12:01 AM, Gary Poster wrote: > I have re-released zc.buildout 1.4.3 as 1.5.0b2, and re-released zc.recipe.egg 1.2.2 as 1.2.3b2. ?My tests seem to show everything back to normal--I was able to build Chris's initial problem example as a smoketest. > > Next week I'll announce betas *not* uploaded to PyPI. ?These will solve the problems identified today. ?I'll provide instructions on how to download and install them locally for tests. > > I appreciate your feedback today, and will appreciate it even more when you try out those betas next week! > > In closing, I'll summarize the problems identified today with 1.5.0b1, with their statuses. > > - virtualenv + zc.buildout fails. ?1.5.0b1 expected the -S of executables to work properly, and it doesn't for virtualenv. ?This may not be fixable within virtualenv, per Carl's email. ?I have a change to zc.buildout that I think will work around this in a local branch. ?Tests fail, so it needs more work. ?This is the most critical problem identified. > > - zc.buildout's bootstrap honors more options in the .cfg file, which caused Jonathan some grief. ?I'm classifying this as a "Won't Fix" unless I get pushback, per my earlier mail. > > - If you have buildout installed in your system Python's site-packages, bootstrap falls over. ?I have not duped this but I think I know the cause. ?I'll try to dupe, and ask Rodrigo to confirm a fix when I'm ready next week. > > - zc.recipe.testing's tests showed some API calls to zc.buildout that failed mysteriously. ?I have not investigated that yet. ?I know of several other existing recipes that use zc.buildout's API that work fine. ?I'm hopeful that this will be easy to diagnose and address. > > - Tarek raised the problem that we can't easily handle beta packages of critical packaging infrastructure. ?Jim countered with information on how buildout already does handle it. ?However, I don't think that this helps with bootstrapping. ?Moreover, it requires explicit configuration to get some of the behavior Jim describes. ?I plan to address Tarek's concerns for now by having betas available through alternate means than PyPI. ?However, this will require more active participation from you all to help prepare a smooth release on PyPI. Thanks for your work ! for "beta" releases, you could use the "dev" tag, and leave the code in the svn trunk. If the metadata of zc.buildout or properly set, we can install the trunk like this: $ easy_install zc.buildout == dev in order to make this work, we need to add somewhere in the documentation (long_description) a link to the svn trunk, followed by an egg fragment marker, and release the metadata with it: http://svn.zope.org/repos/main/zc.buildout/trunk#egg=zc.buildout-dev This will be scanned by easy_install and recognized as the dev version. Pip is not compatible with this feature IIRC. The bootstrap script should also work with this. People can force the dev version in their buildout config as well, if the latest metadata at PyPI has this dev marker. This will make it easier for you to ask people to try the trunk before you release it, Regards Tarek > > Thank you all for your help. ?Talk to you next week. > > Gary > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Sat May 1 10:35:41 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 1 May 2010 10:35:41 +0200 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: On Fri, Apr 30, 2010 at 11:17 PM, Jim Fulton wrote: [..] >> There's a big problem in our ecosystem: ?PyPI + our various installers >> don't have the concept of "beta" release. Any new release is >> considered to be the latest release. > > buildout does. :) > > buildout has a prefer-final option. > > Including: > > ?[buildout] > ?prefer-final = true > > Will cause buildout to only use final release unless there are > no final releases that satisfy a requirement. > > Using this option will cause buildout to automatically downgrade > itself if it previously upgraded to a non-final release. In buildout 2, > prefer-final true will be the default. Ah, great ! on PyPI side, we will be able to mark releases as being beta thanks to PEP 386. Maybe the simple index could differentiate them somehow. [..] >> Last but not least, I think this auto-upgrade feature zc.buildout >> should be removed. ?I'd be in favor of an explicit update of this >> software, rather than having zc.buildout auto-upgrading itself like >> this. > > I find auto update to be useful, but I wouldn't object to a manual > command: > > ?bin/buildout upgrade > > It would be easier to implement than what we have now. +1. An explicit upgrade step would be perfect: people won't have to prevent auto-upgrades by adding options. I am wondering how hard it would be to apply the same principal to all eggs in the buildout, but maybe this is a bit out of scope Tarek -- Tarek Ziad? | http://ziade.org From merwok at netwok.org Sat May 1 12:53:54 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 01 May 2010 12:53:54 +0200 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: <4BDC0842.7000901@netwok.org> > Ah, great ! on PyPI side, we will be able to mark releases as being > beta thanks to PEP 386. > Maybe the simple index could differentiate them somehow. Perhaps hide them, just like older releases are hidden unless you call e.g. XML-RPC method release_packages with show_all set to True. Cheers From manlio.perillo at gmail.com Sat May 1 14:41:10 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Sat, 01 May 2010 14:41:10 +0200 Subject: [Distutils] problems with namespace package Message-ID: <4BDC2166.7050100@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. I have a package named A, with a subpackage B, and I want to create a namespace package A.B.C. In the namespace package setup file I set namespace_packages=['A', 'A.B'], and I use declare_namespace in both A/__init__.py and A/B/__init__.py modules. However if I: 1) install A package 2) install A.B.C package importing A.B.C will raise an ImportError exception. If I reinstall A package, then it seems to work. Where is the problem? Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvcFS0ACgkQscQJ24LbaURdmACgiPqNL/czwKoiFsFIv+41WWQI emEAnjp/a6ZtVB92GkpcONNtOaVFkeir =gKzt -----END PGP SIGNATURE----- From regebro at gmail.com Sat May 1 14:47:28 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 1 May 2010 14:47:28 +0200 Subject: [Distutils] problems with namespace package In-Reply-To: <4BDC2166.7050100@gmail.com> References: <4BDC2166.7050100@gmail.com> Message-ID: On Sat, May 1, 2010 at 14:41, Manlio Perillo wrote: > I have a package named A, with a subpackage B, and I want to create a > namespace package A.B.C. You can't, afaik, not easily at least. You can however make a package that monkeypatches in C into A.B. But that's not really namespace packages. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From jim at zope.com Sat May 1 17:27:30 2010 From: jim at zope.com (Jim Fulton) Date: Sat, 1 May 2010 11:27:30 -0400 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: On Sat, May 1, 2010 at 4:35 AM, Tarek Ziad? wrote: ... > An explicit upgrade step would be perfect: people won't have to > prevent auto-upgrades > by adding options. I am wondering how hard it would be to apply the > same principal > to all eggs in the buildout, but maybe this is a bit out of scope It's easy. Run buildout with -N or use: [buildout] newest = False And, of course, buildout also let's you specify specific versions for projects, via either project requirements or a versions section. Jim -- Jim Fulton From carl at meyerloewen.net Sat May 1 18:18:47 2010 From: carl at meyerloewen.net (Carl Meyer) Date: Sat, 01 May 2010 12:18:47 -0400 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: <4D19E04C-A053-497E-9473-459EB1B65D9C@canonical.com> Message-ID: <4BDC5467.9060609@meyerloewen.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Tarek, Tarek Ziad? wrote: > $ easy_install zc.buildout == dev > > in order to make this work, we need to add somewhere in the > documentation (long_description) a link to the svn trunk, followed by > an egg fragment marker, and release the metadata with it: > > http://svn.zope.org/repos/main/zc.buildout/trunk#egg=zc.buildout-dev > > This will be scanned by easy_install and recognized as the dev > version. Pip is not compatible with this feature IIRC. Pip works great with this feature. I use it regularly. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvcVGcACgkQFRcxmeyPUXLhEACfRtZyYqFxF0l/RIrICxVV31gn 7wUAn3a0qDtBTnE2GfuWM/dChM5jFCqB =vZfl -----END PGP SIGNATURE----- From ziade.tarek at gmail.com Sat May 1 18:21:00 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 1 May 2010 18:21:00 +0200 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: On Sat, May 1, 2010 at 5:27 PM, Jim Fulton wrote: > On Sat, May 1, 2010 at 4:35 AM, Tarek Ziad? > wrote: > ... >> An explicit upgrade step would be perfect: people won't have to >> prevent auto-upgrades >> by adding options. I am wondering how hard it would be to apply the >> same principal >> to all eggs in the buildout, but maybe this is a bit out of scope > > It's easy. ?Run buildout with -N or use: > > ?[buildout] > ?newest = False > > And, of course, buildout also let's you specify specific versions for > projects, via either project requirements or a versions section. > Preventing any upgrade, should be the default behavior imho, Tarek > Jim > > -- > Jim Fulton > -- Tarek Ziad? | http://ziade.org From tseaver at palladion.com Sat May 1 19:24:09 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 01 May 2010 13:24:09 -0400 Subject: [Distutils] problems with namespace package In-Reply-To: <4BDC2166.7050100@gmail.com> References: <4BDC2166.7050100@gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Manlio Perillo wrote: > Hi. > > I have a package named A, with a subpackage B, and I want to create a > namespace package A.B.C. In your examples, it seems as though you want 'A' and 'A.B' to be namespace packages, but 'A.B.C' is a "normal" package. > In the namespace package setup file I set > namespace_packages=['A', 'A.B'], > > and I use declare_namespace in both A/__init__.py and A/B/__init__.py > modules. Are you sure there is no other code in 'A/__init__.py' and 'A/B/__init__.py' of either package? See: http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages > However if I: > 1) install A package > 2) install A.B.C package > > importing A.B.C will raise an ImportError exception. > > If I reinstall A package, then it seems to work. > > Where is the problem? You might compare your setup.py and __init__.py with those in one of the 'zope.app' packages, which successfully use nested namespace packages. E.g. for zope.app.annotation: http://svn.zope.org/repos/main/zope.app.annotation/trunk/setup.py http://svn.zope.org/repos/main/zope.app.annotation/trunk/src/zope/__init__.py http://svn.zope.org/repos/main/zope.app.annotation/trunk/src/zope/app/__init__.py Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvcY7kACgkQ+gerLs4ltQ7c7wCgp3P40zxnfYDK4p+if1FaP9Fo wgsAn0ud53So7Xkk7U/BbDphYergXNjy =+0mW -----END PGP SIGNATURE----- From sdouche at gmail.com Sun May 2 02:40:06 2010 From: sdouche at gmail.com (Sebastien Douche) Date: Sun, 2 May 2010 02:40:06 +0200 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: On Fri, Apr 30, 2010 at 23:17, Jim Fulton wrote: >> There's a big problem in our ecosystem: ?PyPI + our various installers >> don't have the concept of "beta" release. Any new release is >> considered to be the latest release. > > buildout does. :) > > buildout has a prefer-final option. I think it's another need (buildout vs libs). Another point, with this feature activated, some eggs aren't downloadable (thanks to the stupid name chosen by the developers) -- Sebastien Douche Twitter: http://bit.ly/afkrK (agile, lean, python, open source) From manlio_perillo at libero.it Sun May 2 14:40:15 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sun, 02 May 2010 14:40:15 +0200 Subject: [Distutils] problems with namespace package Message-ID: <4BDD72AF.5040100@libero.it> Hi. I have a package named A, with a subpackage B, and I want to create a namespace package A.B.C. In the namespace package setup file I set namespace_packages=['A', 'A.B'], and I use declare_namespace in both A/__init__.py and A/B/__init__.py modules. However if I: 1) install A package 2) install A.B.C package importing A.B.C will raise an ImportError exception. If I reinstall A package, then it seems to work. Where is the problem? Thanks Manlio From manlio_perillo at libero.it Sun May 2 15:35:30 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sun, 02 May 2010 15:35:30 +0200 Subject: [Distutils] setuptools --tag-revision poposed feature Message-ID: <4BDD7FA2.50806@libero.it> Hi. Currently setuptools only support a ``tag-svn-revision`` option. However I use Mercurial, and I would like to tag hg revisions. Looking at setuptool sources, it should not hard to add a new ``tag-revision`` option, used, as an example: --tag-revision=hg In egg_info command class, then, the tags method could do something like: def tags(self): version = '' if self.tag_build: version+=self.tag_build if self.tag_revision: for ep in iter_entry_points( 'setuptools.tag_revision', self.tag_revision): version += '-r%s' % ep.load()() # Only use first entry point break elif os.path.exists('PKG-INFO'): version += '-r%s' % get_pkg_info_revision() if self.tag_date: import time; version += time.strftime("-%Y%m%d") return version The current ``--tag-svn-revision`` will become an alias for ``--tag-revision=svn``, and support will be directly available in setuptools. I don't know if setuptools is still under development (this topic seems quite confused, for me), but can I write a patch, hoping that it will be accepted? Where should I send the patch? Thanks Manlio From jim at zope.com Sun May 2 17:07:26 2010 From: jim at zope.com (Jim Fulton) Date: Sun, 2 May 2010 11:07:26 -0400 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: On Sat, May 1, 2010 at 8:40 PM, Sebastien Douche wrote: > On Fri, Apr 30, 2010 at 23:17, Jim Fulton wrote: >>> There's a big problem in our ecosystem: ?PyPI + our various installers >>> don't have the concept of "beta" release. Any new release is >>> considered to be the latest release. >> >> buildout does. :) >> >> buildout has a prefer-final option. > > I think it's another need (buildout vs libs). I have no idea what you're trying to say. > Another point, with this > feature activated, some eggs aren't downloadable (thanks to the stupid > name chosen by the developers) Could you be more specific, say with an example? If the prefer-final option is used, buildout should still download a non-final distribution, if no final distributions are available. Jim -- Jim Fulton From tseaver at palladion.com Sun May 2 19:06:32 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 02 May 2010 13:06:32 -0400 Subject: [Distutils] setuptools --tag-revision poposed feature In-Reply-To: <4BDD7FA2.50806@libero.it> References: <4BDD7FA2.50806@libero.it> Message-ID: <4BDDB118.9020806@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Manlio Perillo wrote: > Hi. > > Currently setuptools only support a ``tag-svn-revision`` option. > However I use Mercurial, and I would like to tag hg revisions. > > > Looking at setuptool sources, it should not hard to add a new > ``tag-revision`` option, used, as an example: > > --tag-revision=hg > > In egg_info command class, then, the tags method could do something like: > > def tags(self): > version = '' > if self.tag_build: > version+=self.tag_build > if self.tag_revision: > for ep in iter_entry_points( > 'setuptools.tag_revision', self.tag_revision): > version += '-r%s' % ep.load()() > > # Only use first entry point > break > elif os.path.exists('PKG-INFO'): > version += '-r%s' % get_pkg_info_revision() > if self.tag_date: > import time; version += time.strftime("-%Y%m%d") > return version > > > The current ``--tag-svn-revision`` will become an alias for > ``--tag-revision=svn``, and support will be directly available in > setuptools. > > > > I don't know if setuptools is still under development (this topic seems > quite confused, for me), but can I write a patch, hoping that it will be > accepted? > Where should I send the patch? The setuptools tracker is for "generic" setuptools issues: http://bugs.python.org/setuptools/ Depending on the implementation, you might need to patch the plugin used to get setuptools to recognize your "under versino control" files with Mercurial, e.g. http://pypi.python.org/pypi/hg.setuptools or: http://bitbucket.org/jezdez/setuptools_hg/wiki/Home Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvdsRgACgkQ+gerLs4ltQ4M7QCeNCKXAGO9fX1qq8yFPfcma48e 750An16rdD9w5IPLzff0aIh/S6C2Js9y =Ifs0 -----END PGP SIGNATURE----- From manlio_perillo at libero.it Sun May 2 21:13:46 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sun, 02 May 2010 21:13:46 +0200 Subject: [Distutils] setuptools --tag-revision poposed feature In-Reply-To: <4BDDB118.9020806@palladion.com> References: <4BDD7FA2.50806@libero.it> <4BDDB118.9020806@palladion.com> Message-ID: <4BDDCEEA.1060508@libero.it> Tres Seaver ha scritto: > Manlio Perillo wrote: >> Hi. > >> Currently setuptools only support a ``tag-svn-revision`` option. >> However I use Mercurial, and I would like to tag hg revisions. > [...] > >> I don't know if setuptools is still under development (this topic seems >> quite confused, for me), but can I write a patch, hoping that it will be >> accepted? >> Where should I send the patch? > > The setuptools tracker is for "generic" setuptools issues: > > http://bugs.python.org/setuptools/ > It seems there is a problem: Sun May 2 19:09:07 2010: An error occurred. Please check the server log for more infomation. > Depending on the implementation, you might need to patch the plugin used > to get setuptools to recognize your "under versino control" files with > Mercurial, e.g. > > http://pypi.python.org/pypi/hg.setuptools > > or: > > http://bitbucket.org/jezdez/setuptools_hg/wiki/Home > The patch is only required in order to implement an entry point for setuptools.tag_revision, in addition to setuptools.file_finders By the way: why there are two packages that do the same thing? Thanks Manlio From tseaver at palladion.com Sun May 2 21:52:04 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 02 May 2010 15:52:04 -0400 Subject: [Distutils] setuptools --tag-revision poposed feature In-Reply-To: <4BDDCEEA.1060508@libero.it> References: <4BDD7FA2.50806@libero.it> <4BDDB118.9020806@palladion.com> <4BDDCEEA.1060508@libero.it> Message-ID: <4BDDD7E4.9060302@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Manlio Perillo wrote: > Tres Seaver ha scritto: >> Manlio Perillo wrote: >>> Hi. >>> Currently setuptools only support a ``tag-svn-revision`` option. >>> However I use Mercurial, and I would like to tag hg revisions. >> [...] >> >>> I don't know if setuptools is still under development (this topic seems >>> quite confused, for me), but can I write a patch, hoping that it will be >>> accepted? >>> Where should I send the patch? >> The setuptools tracker is for "generic" setuptools issues: >> >> http://bugs.python.org/setuptools/ >> > > It seems there is a problem: > Sun May 2 19:09:07 2010: An error occurred. Please check the server log > for more infomation. Hmm, must be a transient failure: I was able to reach the site just now. >> Depending on the implementation, you might need to patch the plugin used >> to get setuptools to recognize your "under versino control" files with >> Mercurial, e.g. >> >> http://pypi.python.org/pypi/hg.setuptools >> >> or: >> >> http://bitbucket.org/jezdez/setuptools_hg/wiki/Home >> > > The patch is only required in order to implement an entry point for > setuptools.tag_revision, in addition to setuptools.file_finders > > By the way: why there are two packages that do the same thing? "Convergent evolution," I would guess. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvd198ACgkQ+gerLs4ltQ4hfQCgmW+1WjQjS1xfSpPSmZIgLa73 VOwAn1NCtdUcYZ4Tlov0r+W9Rubp4XuV =5ysX -----END PGP SIGNATURE----- From techtonik at gmail.com Sun May 2 22:49:35 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 2 May 2010 23:49:35 +0300 Subject: [Distutils] stdeb in Ubuntu 10.04 Message-ID: There is nothing critical I guess, but stdeb on Ubuntu 10.04 gives a lot of warnings about unknown "buildsystem" option and some others. Any hints why? Should/could these be fixed? pydoctor$ python setup.py --command-packages=stdeb.command bdist_deb .. Unknown option: buildsystem dh_testdir: warning: ignored unknown options in DH_OPTIONS .. dh_installinit Duplicate specification "O=s" for option "O" .. dpkg-deb: warning: 'debian/python-pydoctor/DEBIAN/control' contains user-defined field 'Python-Version' dpkg-deb: building package `python-pydoctor' in `../python-pydoctor_0.3-1_all.deb'. dpkg-deb: warning: ignoring 1 warnings about the control file(s) -- anatoly t. From manlio_perillo at libero.it Mon May 3 02:06:18 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 03 May 2010 02:06:18 +0200 Subject: [Distutils] problems with namespace package Message-ID: <4BDE137A.3000007@libero.it> Sorry for this reply, but it seems I have some problems at receiving mails from my gmail account. I have done some more tests and: * When installing in the system default site directory, namespace package work as expected * When installing in a virtual environment (virtualenv --no-site-packages), the namespace package is not found; it seems to be completely ignored Can someone else confirm that namespace packages do not work with virtualenv? Thanks Manlio From carl at oddbird.net Mon May 3 05:01:45 2010 From: carl at oddbird.net (Carl Meyer) Date: Sun, 02 May 2010 23:01:45 -0400 Subject: [Distutils] problems with namespace package In-Reply-To: <4BDE137A.3000007@libero.it> References: <4BDE137A.3000007@libero.it> Message-ID: <4BDE3C99.7020304@oddbird.net> Manlio Perillo wrote: > Can someone else confirm that namespace packages do not work with > virtualenv? I can confirm the contrary; I use namespace packages regularly with virtualenv and have not had any problems. Carl From pje at telecommunity.com Mon May 3 06:23:03 2010 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 03 May 2010 00:23:03 -0400 Subject: [Distutils] setuptools --tag-revision poposed feature In-Reply-To: <4BDD7FA2.50806@libero.it> References: <4BDD7FA2.50806@libero.it> Message-ID: <20100503042326.C6CBF3A4108@sparrow.telecommunity.com> At 09:35 AM 5/2/2010, Manlio Perillo wrote: >Hi. > >Currently setuptools only support a ``tag-svn-revision`` option. >However I use Mercurial, and I would like to tag hg revisions. > > >Looking at setuptool sources, it should not hard to add a new >``tag-revision`` option, used, as an example: > > --tag-revision=hg > >In egg_info command class, then, the tags method could do something like: > > def tags(self): > version = '' > if self.tag_build: > version+=self.tag_build > if self.tag_revision: > for ep in iter_entry_points( > 'setuptools.tag_revision', self.tag_revision): > version += '-r%s' % ep.load()() > > # Only use first entry point > break This won't work correctly unless you have only one such plugin installed, so the feature needs to be explicitly configured for one specific plugin. > elif os.path.exists('PKG-INFO'): > version += '-r%s' % get_pkg_info_revision() > if self.tag_date: > import time; version += time.strftime("-%Y%m%d") > return version > > >The current ``--tag-svn-revision`` will become an alias for >``--tag-revision=svn``, and support will be directly available in >setuptools. > > > >I don't know if setuptools is still under development (this topic seems >quite confused, for me), but can I write a patch, hoping that it will be >accepted? >Where should I send the patch? Attach it to this outstanding feature request: http://bugs.python.org/setuptools/issue42 As you'll see, there's been some previous discussion for this, and I do plan to make pluggable revision support available in 0.7. It will probably be similar to the most recent patch attached to that issue, with a change to the option name and a few other tweaks. From strawman at astraw.com Mon May 3 06:45:51 2010 From: strawman at astraw.com (Andrew Straw) Date: Mon, 03 May 2010 06:45:51 +0200 Subject: [Distutils] [issue1054967] bdist_deb - Debian packager In-Reply-To: <1272569036.34.0.0721961314704.issue1054967@psf.upfronthosting.co.za> References: <1272569036.34.0.0721961314704.issue1054967@psf.upfronthosting.co.za> Message-ID: <4BDE54FF.6030204@astraw.com> I am moving the venue for this discussion on stdeb to the distutils-sig (from http://bugs.python.org/issue1054967 ). I think this is better discussed here. On 4/29/10 9:23 PM, Barry A. Warsaw wrote: > Manual moving/copying is what I want to avoid. I'd be totally happy with not reinventing the wheel if you think this would be easy to add to stdeb! Specifically: just create debian/ in place and do not build package. > OK, I just added a new command "debianize" to stdeb (git commit 3cac5533 on github). This is currently in the "debianize" branch, which I intend to merge into the master branch after a bit more review. > I have a few other minor suggestions, though if you think this isn't the best place to track them, please let me know where to submit them. > In general, the place is "Issues" at http://github.com/astraw/stdeb , but I've addressed all the issues below. > * Add setup.py entry_points so you don't have to use --command-packages > This (and all other dependencies on setuptools) was removed at the request of the distutils maintainer Tarek Ziad?, who suggested that depending on setuptools prevented listing stdeb in the stdlib distutils documentation. "--command-packages" is standard, documented distutils, and works fine in my experience. If it's convenience you're after, you can add this option to your ~/.pydistutils.cfg file as documented in the stdeb README.rst file. I'm not planning on re-introducing the old behavior. > * Package: shouldn't change dots to dashes. E.g. flufl.enum should be called python-flufl.enum > This should be fixed in git commit 24085bb. > * Update to Standards-Version: 3.8.3 > Thanks for the alert. I'll have a look. Added to the tracker at http://github.com/astraw/stdeb/issues/issue/25 > I'm happy to test whatever you come up with. > Great. If you want to write unit testing infrastructure, that'd be great. -Andrew From strawman at astraw.com Mon May 3 06:46:01 2010 From: strawman at astraw.com (Andrew Straw) Date: Mon, 03 May 2010 06:46:01 +0200 Subject: [Distutils] [issue1054967] bdist_deb - Debian packager In-Reply-To: <4BD9E904.3000509@netwok.org> References: <4BD9E904.3000509@netwok.org> Message-ID: <4BDE5509.1060901@astraw.com> I am moving the venue for this discussion on stdeb to the distutils-sig (from http://bugs.python.org/issue1054967 ). I think this is better discussed there. On 4/29/10 10:16 PM, ?ric Araujo wrote: > ?ric Araujo added the comment: > > Andrew, I am uncomfortable with stdeb. (Trying to express respectful > constructive critique, not just picking on your project.) The version in > Debian testing, 0.5.1-1, still requires setuptools (actually only > pkg_resources, which is split into another package in Debian, for space > or politic reasons). I see, thanks for pointing this out. I thought pkg_resources was in the stdlib (or on that track). I will investigate. > The py2dsc script seems to be obsoleted by the > to-be ?debian? command. Can you refer me to the source code and/or documentation for that command? I was not previously aware of it. > The pypi-install script seems dangerous to me, > because downloading unverified software and putting it into the system > seems a rather bad idead (a.k.a. ?sudo ./setup.py install considered > harmful?). > Yes, some might consider it dangerous, although it does of course require root privileges before installing anything system-wide (and only builds the .deb file with user privileges). Nevertheless, it is a useful tool and safer than "sudo python ./setup.py install" because one can always do "sudo dpkg --purge python-some-package" to remove that package again. > python-debian provides useful support code for generating and reading > debian formats. A quick read show the code would benefit from some 2.6 > modernizing (i.e. gain in simplicity and readability by using modern > idioms and stdlib functions). Yes, I'm not particularly proud of the code in stdeb -- please feel free to hack away. Any and all improvements accepted, preferably in bite-sized commits to a fork on github. > Would using this package in stdeb / > distutils2.command.debian instead of shelling out bring anything? > (Ah, I gather this is the debian command you refer to above?) Not knowing that command, I'm not sure what would be gained. If the command depends on debhelper (by shelling out to dh_make), it would remove that dependency. It also brings all the options described in the README.rst file. -Andrew From attilaolah at gmail.com Mon May 3 09:26:58 2010 From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=) Date: Mon, 3 May 2010 09:26:58 +0200 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: On Fri, Apr 30, 2010 at 04:28, Gary Poster wrote: > I have made a beta release of zc.buildout 1.5.0. 1.5.0b1 was working fine for me, but 1.5.0b2 is acting weird; it seems to be requiring the find-links and allowed-hosts options, and after I set it, it fails with some strange error message: me at myhost:~/devel/users$ ./bin/buildout -Nv Develop: '/path/to/devel/users/.' Installing 'z3c.recipe.depgraph'. Picked: z3c.recipe.depgraph = 0.5 Getting required 'tl.eggdeps' required by z3c.recipe.depgraph 0.5. Picked: tl.eggdeps = 0.4 Getting required 'zc.recipe.egg' required by z3c.recipe.depgraph 0.5. Picked: zc.recipe.egg = 1.2.3b1 Installing graph. While: Installing graph. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/path/to/eggs/zc.buildout-1.5.0b2-py2.6.egg/zc/buildout/buildout.py", line 1660, in main getattr(buildout, command)(args) File "/path/to/devel/eggs/zc.buildout-1.5.0b2-py2.6.egg/zc/buildout/buildout.py", line 532, in install installed_files = self[part]._call(recipe.install) File "/path/to/devel/eggs/zc.buildout-1.5.0b2-py2.6.egg/zc/buildout/buildout.py", line 1204, in _call return f() File "/path/to/devel/eggs/z3c.recipe.depgraph-0.5-py2.6.egg/z3c/recipe/depgraph/recipe.py", line 47, in install reqs, ws = self.egg.working_set(['z3c.recipe.depgraph']) File "/path/to/devel/eggs/zc.recipe.egg-1.2.3b1-py2.6.egg/zc/recipe/egg/egg.py", line 98, in working_set **kw) TypeError: install() got an unexpected keyword argument 'include_site_packages' Are there any docs on what I need to change in my buildout.cfg files to make them behave with the upcoming version of zc.buildout? Thanks, Attila From skip at pobox.com Mon May 3 14:25:37 2010 From: skip at pobox.com (skip at pobox.com) Date: Mon, 3 May 2010 07:25:37 -0500 Subject: [Distutils] Alternative to the default easy-install setup? Message-ID: <19422.49345.963532.366698@montanaro.dyndns.org> I suspect I've asked this before here but have forgotten the answer, so I ask again. We currently use easy_install at work to install most/all third-party packages. This causes problems because sys.path winds up organized like this: 3rd party stuff, followed by local stuff, followed by system stuff As the number of third-party packages grows, the startup time for most applications increases significantly because most of the modules and packages of interest are actually in the local stuff or the system stuff. If this was one or two applications running on a single machine it wouldn't be bad. When you add in NFS and try to start several hundred applications across a multitude of hosts it's pretty easy to freeze the NFS server for a non-trivial amount of time and extend startup times significantly. Our current hack is to comment out the import sys; new=... line which easy_install adds to the end of easy-install.pth. That's not perfect, since the next time a third-party package is installed we have to remember to do it again. And again. And again. Eventually, no matter how careful we are about this, someone forgets. There must be something I can do to mitigate this problem. Can all the third-party eggs be bundled into a super package of some kind to minimize all the file stat-ing? Would some other installer (pip? zc.buildout?) do things differently? Have I maybe missed some flags to easy_install which would improve things? Thanks, -- Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/ From skip at pobox.com Mon May 3 14:30:21 2010 From: skip at pobox.com (skip at pobox.com) Date: Mon, 3 May 2010 07:30:21 -0500 Subject: [Distutils] Alternative to the default easy-install setup? Message-ID: <19422.49629.25910.904668@montanaro.dyndns.org> > I suspect I've asked this before here but have forgotten the answer, so I > ask again. And just after sending I found Philip's response from about a year ago: http://mail.python.org/pipermail/distutils-sig/2009-March/011056.html I'll dig into that a bit. Skip From manlio_perillo at libero.it Mon May 3 15:37:51 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 03 May 2010 15:37:51 +0200 Subject: [Distutils] setuptools --tag-revision poposed feature In-Reply-To: <20100503042326.C6CBF3A4108@sparrow.telecommunity.com> References: <4BDD7FA2.50806@libero.it> <20100503042326.C6CBF3A4108@sparrow.telecommunity.com> Message-ID: <4BDED1AF.80500@libero.it> Phillip J. Eby ha scritto: > At 09:35 AM 5/2/2010, Manlio Perillo wrote: >> Hi. >> >> Currently setuptools only support a ``tag-svn-revision`` option. >> However I use Mercurial, and I would like to tag hg revisions. >> >> > [...] >> def tags(self): >> version = '' >> if self.tag_build: >> version+=self.tag_build >> if self.tag_revision: >> for ep in iter_entry_points( >> 'setuptools.tag_revision', self.tag_revision): >> version += '-r%s' % ep.load()() >> >> # Only use first entry point >> break > > This won't work correctly unless you have only one such plugin > installed, so the feature needs to be explicitly configured for one > specific plugin. > Can you give me more details? I not sure to understand why it will not work when there is more than one plugin. > [...] > Attach it to this outstanding feature request: > > http://bugs.python.org/setuptools/issue42 > > As you'll see, there's been some previous discussion for this, and I do > plan to make pluggable revision support available in 0.7. It will > probably be similar to the most recent patch attached to that issue, > with a change to the option name and a few other tweaks. > Ok. The feature is quite similar to what I had in mind. The only difference I can see is that when a plugin return a non empty string, the next plugin is checked. Thanks Manlio From manlio_perillo at libero.it Mon May 3 15:47:19 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 03 May 2010 15:47:19 +0200 Subject: [Distutils] setuptools --tag-revision poposed feature In-Reply-To: <4BDDD7E4.9060302@palladion.com> References: <4BDD7FA2.50806@libero.it> <4BDDB118.9020806@palladion.com> <4BDDCEEA.1060508@libero.it> <4BDDD7E4.9060302@palladion.com> Message-ID: <4BDED3E7.40105@libero.it> Tres Seaver ha scritto: > [...] >> It seems there is a problem: >> Sun May 2 19:09:07 2010: An error occurred. Please check the server log >> for more infomation. > > Hmm, must be a transient failure: I was able to reach the site just now. > http://bugs.python.org/setuptools/issue42 GET /setuptools/issue42 HTTP/1.1 Host: bugs.python.org User-Agent: Mozilla/5.0 (X11; U; Linux i686; it; rv:1.9.1.9) Gecko/20100414 Iceweasel/3.5.9 (like Firefox/3.5.9) Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: it-it,it;q=0.8,en-us;q=0.5,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive HTTP/1.1 400 Bad Request Date: Mon, 03 May 2010 13:17:30 GMT Server: BaseHTTP/0.3 Python/2.5.2 Content-Type: text/html Via: 1.1 bugs.python.org Connection: close Transfer-Encoding: chunked This happens with both Firefox and Opera. It does not happen using Epiphany. Sure it is strange. Manlio From gary.poster at canonical.com Mon May 3 15:58:47 2010 From: gary.poster at canonical.com (Gary Poster) Date: Mon, 3 May 2010 09:58:47 -0400 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: References: Message-ID: <848D4C22-98EB-4ACD-A607-8DDDEC7F2DB1@canonical.com> On May 3, 2010, at 3:26 AM, Attila Ol?h wrote: > On Fri, Apr 30, 2010 at 04:28, Gary Poster wrote: >> I have made a beta release of zc.buildout 1.5.0. > > 1.5.0b1 was working fine for me, Great. > but 1.5.0b2 is acting weird; 1.5.0b2 is a re-release of 1.4.3 because of issues raised and discussed on this list on Friday. Almost certainly you have some 1.5.0b1 bits still hanging around confusing the 1.4.3/1.5.0b2 code. I suspect that if you get a fresh version of your branch again you'll be fine. Alternatively, try cleaning out the buildout stuff (this usually works for me: ``rm -rf .installed.cfg bin parts``) and then rerun bootstrap and bin/buildout. Gary From attilaolah at gmail.com Mon May 3 16:13:09 2010 From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=) Date: Mon, 3 May 2010 16:13:09 +0200 Subject: [Distutils] Beta release of zc.buildout 1.5.0 In-Reply-To: <848D4C22-98EB-4ACD-A607-8DDDEC7F2DB1@canonical.com> References: <848D4C22-98EB-4ACD-A607-8DDDEC7F2DB1@canonical.com> Message-ID: On Mon, May 3, 2010 at 15:58, Gary Poster wrote: > Alternatively, try cleaning out the buildout stuff (this usually works for me: ``rm -rf .installed.cfg bin parts``) and then rerun bootstrap and bin/buildout. Yep, that helped. Thanks for helping out. And sorry for making noise. Attila From gary.poster at canonical.com Mon May 3 16:56:09 2010 From: gary.poster at canonical.com (Gary Poster) Date: Mon, 3 May 2010 10:56:09 -0400 Subject: [Distutils] zc.buildout 1.5.0b1 and zc.recipe.egg.1.2.3b1 In-Reply-To: <4BDBA64A.8080300@krondo.com> References: <4BDBA64A.8080300@krondo.com> Message-ID: <22E93563-AC6F-4FC0-90DA-DABC51487B73@canonical.com> On Apr 30, 2010, at 11:55 PM, Dave Peticolas wrote: > Thanks for the quick resolution of this, I just wanted to mention > a problem I ran into with these that I don't think I saw reported > in the archives. > > After working around the 'cannot import warnings' problem, I was > getting errors where buildout complained about a missing find-links > parameter. I set that to the empty value: > > [buildout] > find-links = > > And then got the same thing with another one, can't remember which. > I set that to the empty value, and then ran into a traceback about > a parameter, I think it was 'include_site_packages' being passed to > the working_set function that it didn't support. Sorry I can't be > more specific about that, I was rebuilding my virtualenv several > times, when it started working again thanks to the b2 releases. > > Since this was around the time the b2 versions were released, > maybe it was a version mismatch between a b1 and b2 version? This is identical to the problem Attila brought up today. For convenience, I'll repeat what I told him rather than pointing you to the archives. Almost certainly you have some 1.5.0b1 bits still hanging around confusing the 1.4.3/1.5.0b2 code. I suspect that if you get a fresh version of your branch again you'll be fine. Alternatively, try cleaning out the buildout stuff (this usually works for me: ``rm -rf .installed.cfg bin parts``) and then rerun bootstrap and bin/buildout. Gary From dave at krondo.com Mon May 3 17:01:13 2010 From: dave at krondo.com (Dave Peticolas) Date: Mon, 03 May 2010 08:01:13 -0700 Subject: [Distutils] zc.buildout 1.5.0b1 and zc.recipe.egg.1.2.3b1 In-Reply-To: <22E93563-AC6F-4FC0-90DA-DABC51487B73@canonical.com> References: <4BDBA64A.8080300@krondo.com> <22E93563-AC6F-4FC0-90DA-DABC51487B73@canonical.com> Message-ID: <4BDEE539.9040802@krondo.com> On 05/03/2010 07:56 AM, Gary Poster wrote: > > On Apr 30, 2010, at 11:55 PM, Dave Peticolas wrote: > >> Thanks for the quick resolution of this, I just wanted to mention >> a problem I ran into with these that I don't think I saw reported >> in the archives. >> >> After working around the 'cannot import warnings' problem, I was >> getting errors where buildout complained about a missing find-links >> parameter. I set that to the empty value: >> >> [buildout] >> find-links = >> >> And then got the same thing with another one, can't remember which. >> I set that to the empty value, and then ran into a traceback about >> a parameter, I think it was 'include_site_packages' being passed to >> the working_set function that it didn't support. Sorry I can't be >> more specific about that, I was rebuilding my virtualenv several >> times, when it started working again thanks to the b2 releases. >> >> Since this was around the time the b2 versions were released, >> maybe it was a version mismatch between a b1 and b2 version? > > This is identical to the problem Attila brought up today. For convenience, I'll repeat what I told him rather than pointing you to the archives. > > Almost certainly you have some 1.5.0b1 bits still hanging around confusing the 1.4.3/1.5.0b2 code. > > I suspect that if you get a fresh version of your branch again you'll be fine. > > Alternatively, try cleaning out the buildout stuff (this usually works for me: ``rm -rf .installed.cfg bin parts``) and then rerun bootstrap and bin/buildout. I understand, thanks for the explanation. dave From pje at telecommunity.com Mon May 3 18:56:33 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 03 May 2010 12:56:33 -0400 Subject: [Distutils] setuptools --tag-revision poposed feature In-Reply-To: <4BDED1AF.80500@libero.it> References: <4BDD7FA2.50806@libero.it> <20100503042326.C6CBF3A4108@sparrow.telecommunity.com> <4BDED1AF.80500@libero.it> Message-ID: <20100503165625.991953A4119@sparrow.telecommunity.com> At 03:37 PM 5/3/2010 +0200, Manlio Perillo wrote: >Phillip J. Eby ha scritto: > > At 09:35 AM 5/2/2010, Manlio Perillo wrote: > >> Hi. > >> > >> Currently setuptools only support a ``tag-svn-revision`` option. > >> However I use Mercurial, and I would like to tag hg revisions. > >> > >> > > [...] > >> def tags(self): > >> version = '' > >> if self.tag_build: > >> version+=self.tag_build > >> if self.tag_revision: > >> for ep in iter_entry_points( > >> 'setuptools.tag_revision', self.tag_revision): > >> version += '-r%s' % ep.load()() > >> > >> # Only use first entry point > >> break > > > > This won't work correctly unless you have only one such plugin > > installed, so the feature needs to be explicitly configured for one > > specific plugin. > > > >Can you give me more details? >I not sure to understand why it will not work when there is more than >one plugin. >... > > Attach it to this outstanding feature request: > > > > http://bugs.python.org/setuptools/issue42 > > > > As you'll see, there's been some previous discussion for this, and I do > > plan to make pluggable revision support available in 0.7. It will > > probably be similar to the most recent patch attached to that issue, > > with a change to the option name and a few other tweaks. > > > >Ok. >The feature is quite similar to what I had in mind. > >The only difference I can see is that when a plugin return a non empty >string, the next plugin is checked. Hm. I thought you were misreading the patch, but then I looked more closely and saw that I was actually misreading *your* patch... and the problem I thought it had does not actually exist. Sorry for the confusion. From kb at retailarchitects.com Tue May 4 14:36:26 2010 From: kb at retailarchitects.com (Kent) Date: Tue, 4 May 2010 05:36:26 -0700 (PDT) Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies Message-ID: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> Something seems to have changed with a recent release of setuptools. I recently upgraded to -0.6c11 from c9. Under c9, my setup.cfg could specify this: [easy_install] find_links = ../../thirdparty in order to direct setuptools to install from the ../../thirdparty directory instead of searching the internet. This worked for my project's dependencies and the dependencies' dependencies. Now, under c11, settings in the setup.cfg only pass to my project's direct dependencies (install_requires=[] list). They no longer are passed to those dependencies' dependencies. What happened? I've googled and googled and am having no luck. Is my assertion correct? Thanks to anyone who can help. P.S. I also can't find an official PEAK software setuptools forum... is this the best forum to ask this question? From pje at telecommunity.com Tue May 4 20:30:59 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 04 May 2010 14:30:59 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegro ups.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> Message-ID: <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> At 05:36 AM 5/4/2010 -0700, Kent wrote: >Something seems to have changed with a recent release of setuptools. >I recently upgraded to -0.6c11 from c9. > >Under c9, my setup.cfg could specify this: > >[easy_install] >find_links = ../../thirdparty > >in order to direct setuptools to install from the ../../thirdparty >directory instead of searching the internet. This worked for my >project's dependencies and the dependencies' dependencies. > >Now, under c11, settings in the setup.cfg only pass to my project's >direct dependencies (install_requires=[] list). > >They no longer are passed to those dependencies' dependencies. > >What happened? I've googled and googled and am having no luck. Is my >assertion correct? I need more information than this to understand what you mean. Are you talking about running from "setup.py install", or using "easy_install myproject"? What exactly are the dependencies, and the contents of the thirdparty directory? >P.S. I also can't find an official PEAK software setuptools forum... >is this the best forum to ask this question? From http://peak.telecommunity.com/DevCenter/setuptools#mailing-list-and-bug-tracker : "Please use the distutils-sig mailing list for questions and discussion about setuptools" From kb at retailarchitects.com Tue May 4 21:42:02 2010 From: kb at retailarchitects.com (Kent) Date: Tue, 4 May 2010 12:42:02 -0700 (PDT) Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> Message-ID: I am building a turbogears project application. My setup.py looked something like this: ============================================================== # -*- coding: utf-8 -*- try: from setuptools import setup, find_packages except ImportError: from ez_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages setup( name='appname', version='0.1', description='', author='', author_email='', #url='', install_requires=[ #kb start "SQLAlchemy >= 0.6", "tg.devtools >= 2.0.1", "psycopg2 >= 2.0.11", "TurboGears2 >= 2.0.1.1", "Catwalk >= 2.0.2", "Babel >=0.9.4", #can be removed iif use_toscawidgets = False "toscawidgets >= 0.9.7.1", "zope.sqlalchemy >= 0.4 ", "repoze.tm2 >= 1.0a4", "xmllayout >= 0.3", "repoze.what-quickstart >= 1.0", "sprox >= 0.6.6", "BeautifulSoup", "fixture" #kb stop ], setup_requires=["PasteScript >= 1.7"], paster_plugins=['PasteScript', 'Pylons', 'TurboGears2', 'tg.devtools'], packages=find_packages(exclude=['ez_setup']), include_package_data=True, test_suite='nose.collector', tests_require=['WebTest', 'BeautifulSoup'], package_data={'pylotengine': ['i18n/*/LC_MESSAGES/*.mo', 'templates/*/*', 'public/*/*']}, message_extractors={'pylotengine': [ ('**.py', 'python', None), ('templates/**.mako', 'mako', None), ('templates/**.html', 'genshi', None), ('public/**', 'ignore', None)]}, entry_points=""" [paste.app_factory] main = pylotengine.config.middleware:make_app [paste.app_install] main = pylons.util:PylonsInstaller """, ) ============================================================== Because I want a consistent platform, I save all eggs/tars in a thirdparty/ directory and when someone runs "python setup.py install" or "python setup.py develop" with the above file, I wish for dependencies to be taken *only* from my ../../thirdparty folder, and *not* fetched from the internet. This guarantees no updated thirdparty dependency will be the sole cause of something breaking that was working. To accomplish this, with version c9, I added the following lines to setup.cfg: ------------------------------------------- [easy_install] find_links = ../../thirdparty ------------------------------------------- This used to work until I upgrade setuptools to version c11. Now, when a dependency (for example, "tg.devtools >= 2.0.1", from the above 'install_requires' list) is being installed, it has its own dependencies, such as install_requirements = [ 'TurboGears2 >= 2.0.1', 'sqlalchemy-migrate >= 0.5.1', 'SQLAlchemy >= 0.5', 'repoze.what-quickstart >= 1.0', 'repoze.who >= 1.0.10' ] and each of these can have their own requirements. setuptools used to pass my find_links = ../../thirdparty command down to all these dependencies and their dependencies, such that when looking for 'repoze.who >= 1.0.10' (a dependency listed above, for my "tg.devtools >= 2.0.1" dependency), easy_setup would know to look in "../../thirdparty". In c11, it no longer seems to pass this "find_links = ../../ thirdparty" along to dependencies to search for their dependencies. Now, I've worked around this by placing code in my setup.py that dynamically creates and deletes a '~/.pydistutils.cfg' file. I discovered that what I really wanted was 'index_url' instead of 'find_links', so my '~/.pydistutils.cfg' looks like this: [easy_install] index_url = allow_hosts = None This seems to work for me, but I was struggling trying to understand a) why this changed and b) why I couldn't find anyone else complaining that it changed. Is there a better way to accomplish this? While dynamically creating a ~/.pydistutils.cfg file works, it seems idealistically the wrong approach... Thanks for your help. On May 4, 2:30?pm, "P.J. Eby" wrote: > At 05:36 AM 5/4/2010 -0700, Kent wrote: > > > > >Something seems to have changed with a recent release of setuptools. > >I recently upgraded to -0.6c11 from c9. > > >Under c9, my setup.cfg could specify this: > > >[easy_install] > >find_links = ../../thirdparty > > >in order to direct setuptools to install from the ../../thirdparty > >directory instead of searching the internet. ?This worked for my > >project's dependencies and the dependencies' dependencies. > > >Now, under c11, settings in the setup.cfg only pass to my project's > >direct dependencies (install_requires=[] list). > > >They no longer are passed to those dependencies' dependencies. > > >What happened? ?I've googled and googled and am having no luck. ?Is my > >assertion correct? > > I need more information than this to understand what you mean. ?Are > you talking about running from "setup.py install", or using > "easy_install myproject"? ?What exactly are the dependencies, and the > contents of the thirdparty directory? > > >P.S. I also can't find an official PEAK software setuptools forum... > >is this the best forum to ask this question? > > ?Fromhttp://peak.telecommunity.com/DevCenter/setuptools#mailing-list-and-b... > : > > "Please use the > distutils-sig > mailing list for questions and discussion about setuptools" > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-... at python.orghttp://mail.python.org/mailman/listinfo/distutils-sig > > -- > You received this message because you are subscribed to the Google Groups "distutils-sig" group. > To post to this group, send email to distutils-sig at googlegroups.com. > To unsubscribe from this group, send email to distutils-sig+unsubscribe at googlegroups.com. > For more options, visit this group athttp://groups.google.com/group/distutils-sig?hl=en. From pje at telecommunity.com Wed May 5 01:51:23 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 04 May 2010 19:51:23 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> Message-ID: <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> At 12:42 PM 5/4/2010 -0700, Kent wrote: >To accomplish this, with version c9, I added the following lines to >setup.cfg: >------------------------------------------- >[easy_install] >find_links = ../../thirdparty >------------------------------------------- Does this still work if you run "easy_install ." instead of "setup.py install"? From chris at simplistix.co.uk Wed May 5 19:18:25 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 05 May 2010 18:18:25 +0100 Subject: [Distutils] problems building extension for Python2.6 using Visual Studio 2008 Message-ID: <4BE1A861.4040608@simplistix.co.uk> Hi All, I'm using: Python 2.6.5 Visual Studio 2008 I'm trying to compile a package (Twisted, as it happens) for Python 2.6 but having no joy: """ Setting environment for using Microsoft Visual Studio 2008 x86 tools. C:\Program Files\Microsoft Visual Studio 9.0\VC\bin>w: W:\>cd Twisted-10.0.0 W:\Twisted-10.0.0>c:\Python26\python.exe setup.py build running build running build_py running build_ext error: Unable to find vcvarsall.bat """ What am I doing wrong? cheers, Chris From chris at simplistix.co.uk Thu May 6 11:12:39 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 06 May 2010 10:12:39 +0100 Subject: [Distutils] [buildout] ${} substitutions don't happen in buildout:extends? Message-ID: <4BE28807.9090707@simplistix.co.uk> Hi All, Here's the scenario; we have per-project buildouts, but where and how people lay out their checkouts and how they access svn (svn:// versus http:// versus https:// externally). Buildout lends itself to this flexibility by having the per-project buildouts use templated values, with those values being specified in ~/.buildout/default.cfg ...except, this doesn't seem to work for buildout:extends. For example, here's a project buildout: [buildout] extends = ${buildout:base_location}/versions/repoze.bfg-1.2-versions.cfg ... And here's what's in ~/.buildout/default.cfg: [buildout] base_location = /Users/chris/LocalSVN/base ...but, the normal ${} substitution doesn't appear to happen in the extends line, so I get the following error: IOError: [Errno 2] No such file or directory: '/Users/chris.withers/LocalSVN/demoapp/${buildout:base_location}/versions/repoze.bfg-1.2-versions.cfg' What am I doing wrong? How can I achieve my aims with buildout 1.4.3 or later? cheers, Chris From ziade.tarek at gmail.com Thu May 6 15:39:21 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 6 May 2010 15:39:21 +0200 Subject: [Distutils] Distutils 1.0a1 released Message-ID: Hello, I've cut a release of Distutils2 (that was planned for May 1st). This release can't be used in production, is under-documented, but has cool isolated features that can be already used in your projects: - features added during the last year, like the "check" command - a find_packages() function. extracted from setuptools but that can take several root paths. - a metadata reader/writer compatible with all metadata flavors (1.0, 1.1, 1.2) - see http://packages.python.org/Distutils2/metadata.html - a version module that implements PEP 376 - no doc yet ;) Also, we have implemented PEP 345 on PyPI side. You can see for example the project urls in a box on the right side of the page. Release schedule: 1.0a2 - 2010-05-28 (Python 2.7rc1 and GSOC starts) 1.0b1 - 2010-06-25 (Python 2.7 final) 1.0b2 - 2010-07-17 (GSOC Mid-term) 1.0c1 - 2010-07-30 (GSOC Ends) 1.0 final - 2010-08-15 Feedback is welcome ! Regards Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Thu May 6 15:42:02 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 6 May 2010 15:42:02 +0200 Subject: [Distutils] Distutils 1.0a1 released In-Reply-To: References: Message-ID: On Thu, May 6, 2010 at 3:39 PM, Tarek Ziad? wrote: > Hello, > > I've cut a release of Distutils2 (that was planned for May 1st). The link : http://pypi.python.org/pypi/Distutils2/1.0a1 [..] > - a version module that implements PEP 376 - no doc yet ;) s/PEP 376/PEP 386 From jim at zope.com Thu May 6 16:55:11 2010 From: jim at zope.com (Jim Fulton) Date: Thu, 6 May 2010 10:55:11 -0400 Subject: [Distutils] [buildout] ${} substitutions don't happen in buildout:extends? In-Reply-To: <4BE28807.9090707@simplistix.co.uk> References: <4BE28807.9090707@simplistix.co.uk> Message-ID: On Thu, May 6, 2010 at 5:12 AM, Chris Withers wrote: > Hi All, > > Here's the scenario; we have per-project buildouts, but where and how people > lay out their checkouts and how they access svn (svn:// versus http:// > versus https:// externally). > > Buildout lends itself to this flexibility by having the per-project > buildouts use templated values, with those values being specified in > ~/.buildout/default.cfg > > ...except, this doesn't seem to work for buildout:extends. > > For example, here's a project buildout: > > [buildout] > extends = ${buildout:base_location}/versions/repoze.bfg-1.2-versions.cfg > ... > > And here's what's in ~/.buildout/default.cfg: > > [buildout] > base_location = /Users/chris/LocalSVN/base > > ...but, the normal ${} substitution doesn't appear to happen in the extends > line, so I get the following error: > > IOError: [Errno 2] No such file or directory: > '/Users/chris.withers/LocalSVN/demoapp/${buildout:base_location}/versions/repoze.bfg-1.2-versions.cfg' > > What am I doing wrong? Nothing. There are certtain stages of processing where variable substitution isn't done. This is, in part because the variable database hasn't been established yet. I think the situation needs to be improved. At least, I think variable substitutions could reflect the database as it exists at the time of a variables use. This would allow extends to use variables defined on the command line, in default.cfg and in files already read. Of course, the restrictions that remain should be clearly documented. > How can I achieve my aims with buildout 1.4.3 or later? I expect this will work correctly in a later version of buildout. :) Jim -- Jim Fulton From ziade.tarek at gmail.com Thu May 6 20:14:05 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 6 May 2010 20:14:05 +0200 Subject: [Distutils] Distutils 1.0a1 released In-Reply-To: <20100506180708.2BE6B3A4070@sparrow.telecommunity.com> References: <20100506180708.2BE6B3A4070@sparrow.telecommunity.com> Message-ID: 2010/5/6 P.J. Eby : > At 03:39 PM 5/6/2010 +0200, Tarek Ziad? wrote: >> >> a find_packages() function. extracted from setuptools but that can >> take several root paths. > > Darn. ?I was really hoping that distutils2 would enforce a *single* root > path for packages. ?package_dir is an abomination that should die, die, die. > ?(IMO, anyway. ;-) ) Oh well, find_package() is just a helper for the package option. The multi-root option is to make it easier to add extra packages (like a third party package incuded in the distro) Now for the package_dir, option, I wouldn't mind removing it. I have always hated it too. If we just kill it, people will just have to provide packages relative to the root, I am +1 to remove package_dir > > -- Tarek Ziad? | http://ziade.org From regebro at gmail.com Thu May 6 20:31:50 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 6 May 2010 20:31:50 +0200 Subject: [Distutils] Distutils 1.0a1 released In-Reply-To: References: <20100506180708.2BE6B3A4070@sparrow.telecommunity.com> Message-ID: On Thu, May 6, 2010 at 20:14, Tarek Ziad? wrote: > I am +1 to remove package_dir It's gonna be a pain it the ass moving all packages from src/, though. ;) Maybe it'll be scriptable. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From carl at oddbird.net Thu May 6 21:01:06 2010 From: carl at oddbird.net (Carl Meyer) Date: Thu, 06 May 2010 15:01:06 -0400 Subject: [Distutils] Distutils 1.0a1 released In-Reply-To: References: <20100506180708.2BE6B3A4070@sparrow.telecommunity.com> Message-ID: <4BE311F2.7070001@oddbird.net> Lennart Regebro wrote: > On Thu, May 6, 2010 at 20:14, Tarek Ziad? wrote: >> I am +1 to remove package_dir > > It's gonna be a pain it the ass moving all packages from src/, though. > ;) Maybe it'll be scriptable. I don't think removing package_dir means enforcing that all packages have to live in distribution root (at least I hope not). I think it just means distutils2's setup.cfg needs a syntax for specifying "packages" that allows specifying a path to the package. One possible such syntax is here: http://bitbucket.org/carljm/sample-distutils2-project/src/tip/setup.cfg Carl From pje at telecommunity.com Thu May 6 21:17:57 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 06 May 2010 15:17:57 -0400 Subject: [Distutils] Distutils 1.0a1 released In-Reply-To: References: <20100506180708.2BE6B3A4070@sparrow.telecommunity.com> Message-ID: <20100506191755.572E33A4070@sparrow.telecommunity.com> At 08:31 PM 5/6/2010 +0200, Lennart Regebro wrote: >On Thu, May 6, 2010 at 20:14, Tarek Ziad? wrote: > > I am +1 to remove package_dir > >It's gonna be a pain it the ass moving all packages from src/, though. Oh, having a src/ or lib/ root isn't a problem, IMO. I just don't think package_dir should allow *multiple* roots. Most people who are actually using that feature (AFAICT) are just confused about how Python packaging is supposed to work, rather than actually needing multiple roots. From aclark at aclark.net Thu May 6 22:38:31 2010 From: aclark at aclark.net (Alex Clark) Date: Thu, 6 May 2010 20:38:31 +0000 (UTC) Subject: [Distutils] buildout 1.5.0b2 problems Message-ID: Hi all, Is it me or is there trouble with the latest Buildout bootstrap file: svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/bootstrap.py I get this on the latest Ubuntu when trying to run it: aclark at ubuntu:~/Developer/plone-site-admin/buildout$ src/python-buildout/python-2.4/bin/python2.4 bootstrap.py > /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(25)?() -> from optparse import OptionParser (Pdb) l 20 $Id$ 21 """ 22 23 import os, shutil, sys, tempfile, textwrap, urllib, urllib2 24 import pdb ; pdb.set_trace() 25 -> from optparse import OptionParser 26 27 if sys.platform == 'win32': 28 def quote(c): 29 if ' ' in c: 30 return '"%s"' % c # work around spawn lamosity on windows (Pdb) 31 else: 32 return c 33 else: 34 quote = str 35 36 # In order to be more robust in the face of system Pythons, we want to 37 # run without site-packages loaded. This is somewhat tricky, in 38 # particular because Python 2.6's distutils imports site, so starting 39 # with the -S flag is not sufficient. However, we'll start with that: 40 if 'site' in sys.modules: 41 # We will restart with python -S. (Pdb) n > /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(27)?() -> if sys.platform == 'win32': (Pdb) > /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(34)?() -> quote = str (Pdb) > /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(40)?() -> if 'site' in sys.modules: (Pdb) > /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(42)?() -> args = sys.argv[:] (Pdb) > /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(43)?() -> args[0:0] = [sys.executable, '-S'] (Pdb) > /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(44)?() -> args = map(quote, args) (Pdb) > /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(45)?() -> os.execv(sys.executable, args) (Pdb) Traceback (most recent call last): File "bootstrap.py", line 23, in ? import os, shutil, sys, tempfile, textwrap, urllib, urllib2 ImportError: No module named shutil But if I revert an older bootstrap.py, the problem goes away. Alex -- Alex Clark ? http://aclark.net Author of Plone 3.3 Site Administration ? http://aclark.net/plone-site-admin From bradallen137 at gmail.com Thu May 6 23:53:12 2010 From: bradallen137 at gmail.com (Brad Allen) Date: Thu, 6 May 2010 16:53:12 -0500 Subject: [Distutils] developing & managing multiple packages with dependencies Message-ID: Hello, I'm looking for case studies or other examples of management of a development/test/build/QA/release process involving lots of Python packages with dependencies. At work currently, we use Hudson for running our tests, and are using it to produce eggs, sdists, and PIP requirements files. It's shaping up to be a nice way to build our packages, but it's about to get a lot bigger with a lot more packages and complex dependencies. We're working on defining processes for our developers and testers, and we're inventing as we go. I have a good idea where we need to go with this, but our management wants to hear how others in the community handle this kind of challenge with releasing multiple dependent packages together, especially in light of workflows made possible by DVCS. (We're using Git) Does anyone have any stories to tell on this front? From a.cavallo at cavallinux.eu Fri May 7 00:51:50 2010 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Thu, 6 May 2010 23:51:50 +0100 Subject: [Distutils] developing & managing multiple packages with dependencies In-Reply-To: References: Message-ID: Without knowing the size (small, medium or large size company), the platforms (win, linux, mac) and the packages (external or internal) is hard to make any sensible reasoning. For what is is worth keep a central repository for dependencies (either external or internally produced) and enforce it with an iron fist or thinks will go pretty quickly out of control (project A depends on project B and project C on project B' but projectB and projectB' cannot be used at the same time). You probably need a plan for enterprise deployment on the site or across sites so installer integration is a deal breaker (especially if many sites/countries are involved in the deployment stage so communication can be difficult). In my experience I avoided anything that relies on setuptools or any magic/clever stuff that replaces a native installing system (rpm, msi, dpkg or pkg). For legal reason and traceability reasons anything that attempt to download or "dinamically" do things is a no option. I hope this helps, Regards, Antonio On 6 May 2010, at 22:53, Brad Allen wrote: > Hello, > > I'm looking for case studies or other examples of management of a > development/test/build/QA/release process involving lots of Python > packages with dependencies. > > At work currently, we use Hudson for running our tests, and are using > it to produce eggs, sdists, and PIP requirements files. It's shaping > up to be a nice way to build our packages, but it's about to get a lot > bigger with a lot more packages and complex dependencies. We're > working on defining processes for our developers and testers, and > we're inventing as we go. > > I have a good idea where we need to go with this, but our management > wants to hear how others in the community handle this kind of > challenge with releasing multiple dependent packages together, > especially in light of workflows made possible by DVCS. (We're using > Git) > > Does anyone have any stories to tell on this front? > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From bradallen137 at gmail.com Fri May 7 04:42:56 2010 From: bradallen137 at gmail.com (Brad Allen) Date: Thu, 6 May 2010 21:42:56 -0500 Subject: [Distutils] developing & managing multiple packages with dependencies In-Reply-To: References: Message-ID: On Thu, May 6, 2010 at 5:51 PM, Antonio Cavallo wrote: > Without knowing the size (small, medium ?or large size company), > the platforms (win, linux, mac) and the packages (external or internal) > is hard to make any sensible reasoning. Well, I am looking for stories here from other organizations as a source of lessons. However, maybe it would help if I spent more time describing what we're trying to accomplish. We're a medium size software product company (over 100 employees), the supported platforms are Linux and Windows servers, and most of our package dependencies are internal. However, we also have third party open source dependencies. > For what is is worth keep a central repository for dependencies (either external or > internally produced) and enforce it with an iron fist or thinks will go pretty > quickly out of control (project A depends on project B and project C on project B' but > projectB and projectB' cannot be used at the same time). With a DVCS it makes sense to have multiple repositories (a repo for each package, er, I mean 'module distribution'), though we do have a centralized workflow with a central server containing all the repos. The approach we're moving toward is to have the 'master' branch of each repo associated with the most current stable release, and a 'development' branch for the most current development work. For old releases of a given package under maintenance, there would be a development and a release branch for each major + minor version number. For example maintenance version 2.7 would have the branches dev_2.7 and rel_2.7. Each development and release branch of each package would have a separate Hudson job running the tests, and building eggs, sdists, and pip requirements files. The script for building the eggs creates a version consisting of the version in setup.py + the tag_build from setup.cfg + the Hudson build number. For release branches, the build script takes the extra step of creating a Git tag of the version (including the Hudson build number), so each build can be linked back to a commit id in the repo. > In my experience I avoided anything that relies on setuptools or any > magic/clever stuff that replaces a native installing system (rpm, msi, dpkg > or pkg). Well, we're pretty much relying on setuptools/buildout, and I don't see anything wrong with that, as long as we have a reasonable migration path distribute/distutils2 in the future. Up till now we've built the eggs manually and put them on our package server, and deliver buildouts to our customers using zc.sourcerelease. Now that we've automated our build process using Hudson the natural next step will be get buildout to make use of the pip requirements file created by the Hudson jobs. The actual release to the customer usually involves a selection of 'top level' packages, and all of them need to rely on the same version of the core library dependencies. In the past, we hand-coded a version.cfg file for use by buildout, and that file was put under version control and tagged for each release. In the future, we're considering hand-coding a pip requirements file to contain the desired version of each package needed by a customer, and setting up a Hudson job to run the tests for that particular set of versions. That would define a 'KGS' (Known Good Set) of top level packages and their dependencies, and we would keep that KGS as a release configuration under version control. > For legal reason and traceability reasons anything that attempt to > download or "dinamically" do things is a no option. I have no clue what you mean by that. What do you have against 'downloading'? From regebro at gmail.com Fri May 7 07:21:41 2010 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 7 May 2010 07:21:41 +0200 Subject: [Distutils] Distutils 1.0a1 released In-Reply-To: <20100506191755.572E33A4070@sparrow.telecommunity.com> References: <20100506180708.2BE6B3A4070@sparrow.telecommunity.com> <20100506191755.572E33A4070@sparrow.telecommunity.com> Message-ID: On Thu, May 6, 2010 at 21:17, P.J. Eby wrote: > Oh, having a src/ or lib/ root isn't a problem, IMO. ?I just don't think > package_dir should allow *multiple* roots. I see. +1 -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From a.cavallo at cavallinux.eu Fri May 7 09:43:18 2010 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Fri, 7 May 2010 08:43:18 +0100 Subject: [Distutils] developing & managing multiple packages with dependencies In-Reply-To: References: Message-ID: >> ith a DVCS it makes sense to have multiple repositories (a repo for > each package, er, I mean 'module distribution'), though we do have a > centralized workflow with a central server containing all the repos. There's nothing specific on the source code control tool: I assume developers will rely anyway on a "frozen" module/package set (or release, or tag or KGS as you named later on). > development and a release branch for each major + minor version > number. For example maintenance version 2.7 would have the branches > dev_2.7 and rel_2.7. For the version (the one exposed through __version__) is it worth to keep it as X.Y.Z or it may break the bdist_msi installer, especially if you're on a multiplatform: I wished they standardised ___version__ to be this way in the language so intra package dependency could be done in a reasonable way. > Well, we're pretty much relying on setuptools/buildout, and I don't > see anything wrong with that, as long as we have a reasonable > migration path distribute/distutils2 In the past (it might be still now) it tried to replace python distribution files (like site.py if I'm not wrong, it was long while ago): so at each python upgrade/patch released (like in security patches) you needed to reinstall it, breaking dependencies. Beside the technical reasons, from a network administration point of view is a bad way to replace a native way to install things. Plenty of company have developers without administrator privileges on a machine. All of a sudden if you are a syadmin and must know what is installed on a remote machine (possible in another continent) you could not use any longer the standard system tools (rpm -qi on linux) but you need to be logged as the developer and go figuring out the way he/she installed things. Again I suggest to stay away from anything that relies on setuptools, that is my professional experience, and I haven't seen any (good) reason to suggest the contrary: you needs might be different. > The actual release to the customer usually involves a selection of > 'top level' packages, and all of them need to rely on the same version > of the core library dependencies. That is what I meant (KGS) at the begin of the message, I think I haven't expressed myself clearly :( If you use hudson (or anything like that) you should have a plan to not hand fiddling with files anyway. > >> For legal reason and traceability reasons anything that attempt to >> download or "dinamically" do things is a no option. > > I have no clue what you mean by that. What do you have against 'downloading'? Traceability is I give you the "product" (in business lingo "deliverable") now the questions are: - What it depends upon? python + module foo version X + module bar version Y and so on. - How do I rebuild it if you company goes bankrupt (I hope not but it is an eventuality)? - Is there any hidden backdoor any of you employee has put in one of the many component? - How do we get the name of the offender if that happen? Now you probably can see why I have a LOT against wild downloading. Beside this there are licensing problems like using GPL code on a commercial software although I must admit python is quite liberal on that point. I hope this helps. Best regards, Antonio From kb at retailarchitects.com Fri May 7 14:54:44 2010 From: kb at retailarchitects.com (Kent) Date: Fri, 7 May 2010 05:54:44 -0700 (PDT) Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> Message-ID: <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> > > Does this still work if you run "easy_install ." instead of "setup.py install"? > No, when I try this it attempts to download from the internet: "Downloading http://pypi.python.org/packages/source/P/Paste/Paste-1.7.3.1.tar.gz#md5=743deb94d3f54ba25522f7d5116a90f8" while processing a dependency's dependency. This happens despite the fact that I have this section in setup.cfg: ========================================= [easy_install] index_url = ../../thirdparty # only allow thirdparty installations, so we don't have version # incompatibilties by downloading from web... allow_hosts = None ========================================= Like I mentioned, I do have a workaround in place that is working for me. I post it here in case it may help others: ________________________________________________________________________________________ ________________________________________________________________________________________ try: from setuptools import setup, find_packages except ImportError: from ez_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages #kb, now create a .pydistutils.cfg file in the home directory # so that we have control over dependencies' dependencies. # for example, when we instead tg.devtools below, we don't # want it to go to the internet either. import os thirdparty_location = os.path.abspath(os.path.dirname(__file__) + '../../thirdparty') cfgfilename=os.path.expanduser('~/.pydistutils.cfg') cfgfile = open(cfgfilename, 'w') cfgfile.write(""" [easy_install] index_url = """ + thirdparty_location + """ allow_hosts = None """) cfgfile.close() setup( name='pylotengine', version='0.1', ... [ reset of setup() function call] ... ) #kb # now remove our .pydistutils.cfg os.unlink(cfgfilename) ________________________________________________________________________________________ ________________________________________________________________________________________ From kb at retailarchitects.com Fri May 7 15:09:00 2010 From: kb at retailarchitects.com (Kent) Date: Fri, 7 May 2010 06:09:00 -0700 (PDT) Subject: [Distutils] inability to pass setup.py command line arguments to dependency setups Message-ID: Consider the case where you want your setup to install third-party software, but you want/need to pass an argument to the command line that runs "python setup.py --argument install" or "python setup.py -- argument bdist_egg" As far as I could research, this feature is unavailable. If there is a graceful way to accomplish this I would really like to know. If there is no way, I offer a workaround to those who require this functionality. Consider a setup.py with the following setup() function call: ________________________________________________________________________________________ setup( name='pylotengine', version='0.1', description='', author='', author_email='', install_requires=[ "SQLAlchemy >= 0.6", "tg.devtools >= 2.0.1", ... ... ) ________________________________________________________________________________________ Now, when SQLAlchemy is to be installed, there is a really nice feature that can be enabled to build it with C extensions for performance, so you want easy_install to run: "python setup.py --with-cextensions bdist_egg" instead of just: "python setup.py bdist_egg" In order to pass the "--with-cextensions" argument, I do the following (again, not a very graceful solution): ________________________________________________________________________________________ # -*- coding: utf-8 -*- try: from setuptools import setup, find_packages except ImportError: from ez_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages #kb: since apparently setuptools has no mechanism to pass # arguments to the dependencies, I am "decorating" the function # that runs the command and adding the arguments myself... from setuptools.command.easy_install import easy_install normal_run_setup_fn = easy_install.run_setup def setup_hook(self, setup_script, setup_base, args): # SQLAlchemy we want to run setup.py --with-cextensions if setup_script.find('/SQLAlchemy') > -1 and setup_script.endswith('setup.py'): args.insert(0,'--with-cextensions') normal_run_setup_fn(self, setup_script, setup_base, args) easy_install.run_setup = setup_hook setup( name='pylotengine', version='0.1', description='', author='', author_email='', #url='', install_requires=[ #kb start "SQLAlchemy >= 0.6", "tg.devtools >= 2.0.1", "psycopg2 >= 2.0.11", ... ...rest of setup() call... ________________________________________________________________________________________ Hope this is helpful to someone... From jbaker at zeomega.com Fri May 7 15:26:18 2010 From: jbaker at zeomega.com (Jason Baker) Date: Fri, 7 May 2010 08:26:18 -0500 Subject: [Distutils] developing & managing multiple packages with dependencies In-Reply-To: References: Message-ID: On Fri, May 7, 2010 at 2:43 AM, Antonio Cavallo wrote: > > Beside the technical reasons, from a network administration point of view > is a bad way to replace a native way to install things. Plenty of company > have developers without administrator privileges on a machine. > > All of a sudden if you are a syadmin and must know what is installed > on a remote machine (possible in another continent) you could not use > any longer the standard system tools (rpm -qi on linux) but you need > to be logged as the developer and go figuring out the way he/she > installed things. > > Again I suggest to stay away from anything that relies on setuptools, that > is my professional experience, and I haven't seen any (good) reason > to suggest the contrary: you needs might be different. > I agree that using native operating system tools to install packages is a good thing. There's one big problem with that though: it becomes near impossible to sandbox packages into their own virtualenvs. This is important to us because we have multiple packages that may rely upon different versions of different packages. > Traceability is I give you the "product" (in business lingo "deliverable") > now the questions are: > - What it depends upon? python + module foo version X + module bar version > Y and so on. > - How do I rebuild it if you company goes bankrupt (I hope not but it is > an eventuality)? > - Is there any hidden backdoor any of you employee has put in one of the > many component? > - How do we get the name of the offender if that happen? > I suppose I can see the benefit of having dependencies in version control. I'm not sure I like the idea of ruling things with an iron fist though. I could see managing dependencies and managing the build process turning into a full time job. I suppose there may be benefits to that, but whether or not we can dedicate a person to it is a decision that is well above me. :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.pradal at cirad.fr Fri May 7 16:01:35 2010 From: christophe.pradal at cirad.fr (Christophe Pradal) Date: Fri, 07 May 2010 16:01:35 +0200 Subject: [Distutils] developing & managing multiple packages with dependencies In-Reply-To: References: Message-ID: <4BE41D3F.4040001@cirad.fr> Le 06/05/2010 23:53, Brad Allen a ?crit : > Hello, > > I'm looking for case studies or other examples of management of a > development/test/build/QA/release process involving lots of Python > packages with dependencies. > Our context is a research teams working togeteher and integrating different packages both in Python and C, C++ and Fortran. We use buildbot for the continuous integration. All the stages (build, installation, test, QA upload) are done using setuptools. Each package is - built (C, C++ and Fortran) using scons integrated with setuptools. - install, test, QA - egg are cretaed with the version (X.Y.Z) and svn revision. It depends on the exact revision number of each its dependencies for binary compatibility. Finally the eggs are upload on a central repository. It is quite inflexible if you use Python packages, but it is needed when you use binary packages with cross dependencies. cheers Christophe > At work currently, we use Hudson for running our tests, and are using > it to produce eggs, sdists, and PIP requirements files. It's shaping > up to be a nice way to build our packages, but it's about to get a lot > bigger with a lot more packages and complex dependencies. We're > working on defining processes for our developers and testers, and > we're inventing as we go. > > I have a good idea where we need to go with this, but our management > wants to hear how others in the community handle this kind of > challenge with releasing multiple dependent packages together, > especially in light of workflows made possible by DVCS. (We're using > Git) > > Does anyone have any stories to tell on this front? > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -- Christophe Pradal INRIA/CIRAD project-team Virtual Plants UMR DAP - D?veloppement et Am?lioration des Plantes TA A96/02, CIRAD, Avenue Agropolis 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 75 63 From gary.poster at canonical.com Fri May 7 17:40:28 2010 From: gary.poster at canonical.com (Gary Poster) Date: Fri, 7 May 2010 11:40:28 -0400 Subject: [Distutils] buildout 1.5.0b2 problems In-Reply-To: References: Message-ID: <118B4166-B78C-45F6-86F5-43E144E1750A@canonical.com> On May 6, 2010, at 4:38 PM, Alex Clark wrote: > Hi all, > > Is it me or is there trouble with the latest Buildout bootstrap file: > svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/bootstrap.py > > I get this on the latest Ubuntu when trying to run it: > > aclark at ubuntu:~/Developer/plone-site-admin/buildout$ src/python-buildout/python-2.4/bin/python2.4 bootstrap.py >> /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(25)?() > -> from optparse import OptionParser > (Pdb) l > 20 $Id$ > 21 """ > 22 > 23 import os, shutil, sys, tempfile, textwrap, urllib, urllib2 > 24 import pdb ; pdb.set_trace() > 25 -> from optparse import OptionParser > 26 > 27 if sys.platform == 'win32': > 28 def quote(c): > 29 if ' ' in c: > 30 return '"%s"' % c # work around spawn lamosity on windows > (Pdb) > 31 else: > 32 return c > 33 else: > 34 quote = str > 35 > 36 # In order to be more robust in the face of system Pythons, we want to > 37 # run without site-packages loaded. This is somewhat tricky, in > 38 # particular because Python 2.6's distutils imports site, so starting > 39 # with the -S flag is not sufficient. However, we'll start with that: > 40 if 'site' in sys.modules: > 41 # We will restart with python -S. > (Pdb) n >> /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(27)?() > -> if sys.platform == 'win32': > (Pdb) >> /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(34)?() > -> quote = str > (Pdb) >> /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(40)?() > -> if 'site' in sys.modules: > (Pdb) >> /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(42)?() > -> args = sys.argv[:] > (Pdb) >> /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(43)?() > -> args[0:0] = [sys.executable, '-S'] > (Pdb) >> /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(44)?() > -> args = map(quote, args) > (Pdb) >> /home/aclark/Developer/plone-site-admin/buildout/bootstrap.py(45)?() > -> os.execv(sys.executable, args) > (Pdb) > Traceback (most recent call last): > File "bootstrap.py", line 23, in ? > import os, shutil, sys, tempfile, textwrap, urllib, urllib2 > ImportError: No module named shutil > > But if I revert an older bootstrap.py, the problem goes away. Hi. Sorry for the late reply: I had some dental surgery this week that put me out of commission for a couple of days. This does not make immediate sense to me, and I can't dupe, trying with Python 2.4, buildout 1.5.0b2, and the trunk bootstrap.py, as shown above. Alex and I are exploring this on IRC. I'll report back if we have a resolution. Thanks Gary From aclark at aclark.net Fri May 7 17:59:40 2010 From: aclark at aclark.net (Alex Clark) Date: Fri, 7 May 2010 15:59:40 +0000 (UTC) Subject: [Distutils] buildout 1.5.0b2 problems References: <118B4166-B78C-45F6-86F5-43E144E1750A@canonical.com> Message-ID: On 2010-05-07, Gary Poster wrote: > Hi. Sorry for the late reply: I had some dental surgery this week that put me out of commission for a couple of days. > > This does not make immediate sense to me, and I can't dupe, trying with Python 2.4, buildout 1.5.0b2, and the trunk bootstrap.py, as shown above. Alex and I are exploring this on IRC. I'll report back if we have a resolution. OK Sorry to implicate Buildout here, I was using the Virtualenv inside Florian's Python buildout and not the "real" Python? PEBKAC Thanks guys > Thanks > > Gary > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Alex Clark ? http://aclark.net Author of Plone 3.3 Site Administration ? http://aclark.net/plone-site-admin From bradallen137 at gmail.com Fri May 7 18:01:27 2010 From: bradallen137 at gmail.com (Brad Allen) Date: Fri, 7 May 2010 11:01:27 -0500 Subject: [Distutils] Distutils 1.0a1 released In-Reply-To: References: Message-ID: On Thu, May 6, 2010 at 8:39 AM, Tarek Ziad? wrote: > - a version module that implements PEP 376 - no doc yet ;) You mean PEP 386, right? From pje at telecommunity.com Fri May 7 18:57:55 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 07 May 2010 12:57:55 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegro ups.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> Message-ID: <20100507165758.ED2A83A4070@sparrow.telecommunity.com> At 05:54 AM 5/7/2010 -0700, Kent wrote: > > > > Does this still work if you run "easy_install ." instead of > "setup.py install"? > > > >No, when I try this it attempts to download from the internet: >"Downloading >http://pypi.python.org/packages/source/P/Paste/Paste-1.7.3.1.tar.gz#md5=743deb94d3f54ba25522f7d5116a90f8" > >while processing a dependency's dependency. > >This happens despite the fact that I have this section in setup.cfg: > >========================================= >[easy_install] >index_url = ../../thirdparty I thought you were using --find-links as of your last email? So, I'm a little confused as to what scenario you're actually trying to fix. The next thing I would do is verify that the config file is actually being read, by setting the DISTUTILS_DEBUG environment variable to "yes", and then running "easy_install -v ." and observing the output. >Like I mentioned, I do have a workaround in place that is working for >me. I post it here in case it may help others: Please don't do that, and *nobody else should do it either*. You are going to ERASE people's personal distutils configuration files with that code! (You're not even checking to see if it exists first!) I would be *extremely* upset if I tried to install your package and it erased one of my .pydistutils.cfg files, with all their command aliases, wikiup configs, path customizations, etc. (Of course, when run under easy_install control, the sandbox should catch your setup script's antics and terminate it with an error without harming any files, but still... it's pure evil and the sort of thing that should NEVER, ever be done in a setup.py, on pain of banishment from PyPI.) From glyph at twistedmatrix.com Fri May 7 19:33:42 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 7 May 2010 13:33:42 -0400 Subject: [Distutils] inability to pass setup.py command line arguments to dependency setups In-Reply-To: References: Message-ID: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> On May 7, 2010, at 9:09 AM, Kent wrote: > Consider the case where you want your setup to install third-party > software, but you want/need to pass an argument to the command line > that runs "python setup.py --argument install" or "python setup.py -- > argument bdist_egg" > > As far as I could research, this feature is unavailable. And for good reason, I should think. I really hope that nobody ever adds this feature. If you require SQLAlchemy to be installed "--with-cextensions", then what happens when your package is installed in an environment that already has SQLAlchemy installed *without* that flag? Does it stomp on the existing installation? What if the user installed one already with that flag and some other flags as well? What if it's system-installed and you're doing a user-install? Basically, compile-time and install-time options are a configuration nightmare. They should represent only how and where a package is installed, not what features it has. The correct solution to your problem would be to get SQLAlchemy to fix its broken deployment setup and split itself into 2 packages, "SQLAlchemyCExtensions" and "SQLAlchemy", and then have your project depend on both, not to try to cram installer options into the dependency language. For confirmation of this theory, you need look no further than the excruciating user-experience of source-based installation systems with 'variant' support, like gentoo's Portage and *BSD's Ports, versus the relatively non-excruciating experience of packaging systems which express compile-time options as different packages like Yum and Apt. From kb at retailarchitects.com Fri May 7 19:36:39 2010 From: kb at retailarchitects.com (Kent Bower) Date: Fri, 07 May 2010 13:36:39 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <20100507165758.ED2A83A4070@sparrow.telecommunity.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> <20100507165758.ED2A83A4070@sparrow.telecommunity.com> Message-ID: <4BE44FA7.20505@retailarchitects.com> An HTML attachment was scrubbed... URL: From kb at retailarchitects.com Fri May 7 19:56:41 2010 From: kb at retailarchitects.com (Kent Bower) Date: Fri, 07 May 2010 13:56:41 -0400 Subject: [Distutils] inability to pass setup.py command line arguments to dependency setups In-Reply-To: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> Message-ID: <4BE45459.7050907@retailarchitects.com> Just because there are configuration problems associated with adding a feature like the one I needed is absolutely no reason to abandon it when it can bring value to the tool if used correctly and in some circumstances. I considered some of those exact complications "what if it was already installed, etc" and with my company's project, where I am using this useful tool in a circumstance you may overlook (it is perfectly acceptable to have such a feature, *despite* the list of complications you mention), such a feature would have been very valuable. Since it is useful in my case, I understand it would be valuable for others as well. (I don't appreciate the aggressive tone of your reply, though, nor do I see how my good faith efforts to help others warrant this... how did I possibly offend you with my post??) On the other hand, I appreciate your "correct solution" as a good approach and I'll forward this idea to an author of SQLAlchemy for his consideration. On 5/7/2010 1:33 PM, Glyph Lefkowitz wrote: > On May 7, 2010, at 9:09 AM, Kent wrote: > > >> Consider the case where you want your setup to install third-party >> software, but you want/need to pass an argument to the command line >> that runs "python setup.py --argument install" or "python setup.py -- >> argument bdist_egg" >> >> As far as I could research, this feature is unavailable. >> > And for good reason, I should think. I really hope that nobody ever adds this feature. > > If you require SQLAlchemy to be installed "--with-cextensions", then what happens when your package is installed in an environment that already has SQLAlchemy installed *without* that flag? Does it stomp on the existing installation? What if the user installed one already with that flag and some other flags as well? What if it's system-installed and you're doing a user-install? > > Basically, compile-time and install-time options are a configuration nightmare. They should represent only how and where a package is installed, not what features it has. The correct solution to your problem would be to get SQLAlchemy to fix its broken deployment setup and split itself into 2 packages, "SQLAlchemyCExtensions" and "SQLAlchemy", and then have your project depend on both, not to try to cram installer options into the dependency language. > > For confirmation of this theory, you need look no further than the excruciating user-experience of source-based installation systems with 'variant' support, like gentoo's Portage and *BSD's Ports, versus the relatively non-excruciating experience of packaging systems which express compile-time options as different packages like Yum and Apt. > > From mike_mp at zzzcomputing.com Fri May 7 20:08:29 2010 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 7 May 2010 14:08:29 -0400 Subject: [Distutils] [sqlalchemy] Re: inability to pass setup.py command line arguments to dependency setups In-Reply-To: <4BE45459.7050907@retailarchitects.com> References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> <4BE45459.7050907@retailarchitects.com> Message-ID: <50C9AE5B-63F5-4A41-B63A-A16DC654D106@zzzcomputing.com> I'd only mention that Storm has a C extension/non C extension flag as well, and only offers one source distribution on Pypi. You have to modify a variable directly within setup.py. Our setup.py features the same capability (its just our C extension is off by default for 0.6 since it was just written, which is the same case for when Storm first introduced its C extension). On May 7, 2010, at 1:56 PM, Kent Bower wrote: > Just because there are configuration problems associated with adding a feature like the one I needed is absolutely no reason to abandon it when it can bring value to the tool if used correctly and in some circumstances. I considered some of those exact complications "what if it was already installed, etc" and with my company's project, where I am using this useful tool in a circumstance you may overlook (it is perfectly acceptable to have such a feature, *despite* the list of complications you mention), such a feature would have been very valuable. > > Since it is useful in my case, I understand it would be valuable for others as well. > > (I don't appreciate the aggressive tone of your reply, though, nor do I see how my good faith efforts to help others warrant this... how did I possibly offend you with my post??) > > On the other hand, I appreciate your "correct solution" as a good approach and I'll forward this idea to an author of SQLAlchemy for his consideration. > > > On 5/7/2010 1:33 PM, Glyph Lefkowitz wrote: >> On May 7, 2010, at 9:09 AM, Kent wrote: >> >> >>> Consider the case where you want your setup to install third-party >>> software, but you want/need to pass an argument to the command line >>> that runs "python setup.py --argument install" or "python setup.py -- >>> argument bdist_egg" >>> >>> As far as I could research, this feature is unavailable. >>> >> And for good reason, I should think. I really hope that nobody ever adds this feature. >> >> If you require SQLAlchemy to be installed "--with-cextensions", then what happens when your package is installed in an environment that already has SQLAlchemy installed *without* that flag? Does it stomp on the existing installation? What if the user installed one already with that flag and some other flags as well? What if it's system-installed and you're doing a user-install? >> >> Basically, compile-time and install-time options are a configuration nightmare. They should represent only how and where a package is installed, not what features it has. The correct solution to your problem would be to get SQLAlchemy to fix its broken deployment setup and split itself into 2 packages, "SQLAlchemyCExtensions" and "SQLAlchemy", and then have your project depend on both, not to try to cram installer options into the dependency language. >> >> For confirmation of this theory, you need look no further than the excruciating user-experience of source-based installation systems with 'variant' support, like gentoo's Portage and *BSD's Ports, versus the relatively non-excruciating experience of packaging systems which express compile-time options as different packages like Yum and Apt. >> >> > > -- > You received this message because you are subscribed to the Google Groups "sqlalchemy" group. > To post to this group, send email to sqlalchemy at googlegroups.com. > To unsubscribe from this group, send email to sqlalchemy+unsubscribe at googlegroups.com. > For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en. > From kb at retailarchitects.com Fri May 7 20:20:51 2010 From: kb at retailarchitects.com (Kent) Date: Fri, 7 May 2010 11:20:51 -0700 (PDT) Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <4BE44FA7.20505@retailarchitects.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> <20100507165758.ED2A83A4070@sparrow.telecommunity.com> <4BE44FA7.20505@retailarchitects.com> Message-ID: <34b00a39-3aa6-4c25-8a2e-5ece9ddac82b@j33g2000yqn.googlegroups.com> > I thought you were using --find-links as of your last email? So, I'm a little confused as to what scenario you're actually trying to fix. > Yes, I mentioned, I *used* to be using --find-links, but I determined this is not what I really wanted (though it did work in version c9). > The next thing I would do is verify that the config file is actually being read, by setting the DISTUTILS_DEBUG environment variable to "yes", and then running "easy_install -v ." and observing the output. > I know the answer is "absolutely, yes" for top level dependencies, as web access is blocked due to the setup.cfg line. Just not for the dependencies' dependencies. When I get a chance and I am in the office, I would gladly post the results of the debug. > >> Like I mentioned, I do have a workaround in place that is working for >> me. I post it here in case it may help others: > > Please don't do that, and *nobody else should do it either*. You are going to ERASE people's personal distutils configuration files with that code! (You're not even checking to see if it exists first!) > Thank you for posting the disclaimer before use. I am exactly aware it is deleting that file and for my project that is completely acceptable, so I did not do anything fancy like backing it up. I'd hope anyone using this would have an understanding of python enough to see what is going on and modify it to meet their needs, but that is a poor assumption so a disclaimer would have been better; thank you for posting such. > I would be *extremely* upset if I tried to install your package and it erased one of my .pydistutils.cfg files, with all their command aliases, wikiup configs, path customizations, etc. > (Again, for my project this is a non-issue, as our company installs our software on our client's VMs - we are in complete control of the environment, so the "pure evil", as you called it isn't at all - it fixes an issue for us.) > (Of course, when run under easy_install control, the sandbox should catch your setup script's antics and terminate it with an error without harming any files, but still... it's pure evil and the sort of thing that should NEVER, ever be done in a setup.py, on pain of banishment from PyPI.) > I can only hope you are saying that tongue-in-cheek, because it is a bit harsh (how did I possibly offend you?). My workaround is acceptable for me, (not for a cheese shop project, I agree), but may help others in my circumstances. setuptools is very useful software even for projects that are never going to be publicly released. Thanks again for your help thus far. From kb at retailarchitects.com Fri May 7 20:44:20 2010 From: kb at retailarchitects.com (Kent) Date: Fri, 7 May 2010 11:44:20 -0700 (PDT) Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <34b00a39-3aa6-4c25-8a2e-5ece9ddac82b@j33g2000yqn.googlegroups.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> <20100507165758.ED2A83A4070@sparrow.telecommunity.com> <4BE44FA7.20505@retailarchitects.com> <34b00a39-3aa6-4c25-8a2e-5ece9ddac82b@j33g2000yqn.googlegroups.com> Message-ID: <1c434699-3586-48d0-a8f3-70d2629e147d@k29g2000yqh.googlegroups.com> > > The next thing I would do is verify that the config file is actually being read, by setting the DISTUTILS_DEBUG environment variable to "yes", and then running "easy_install -v ." and observing the output. I've pasted the output here: http://pastebin.com/1it5XPAL Hopefully you can make sense of the output. Thanks again. From gdementen at gmail.com Fri May 7 20:58:38 2010 From: gdementen at gmail.com (Gaetan de Menten) Date: Fri, 7 May 2010 20:58:38 +0200 Subject: [Distutils] [sqlalchemy] Re: inability to pass setup.py command line arguments to dependency setups In-Reply-To: <4BE45459.7050907@retailarchitects.com> References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> <4BE45459.7050907@retailarchitects.com> Message-ID: FWIW, it is perfectly possible to package the thing separately as Glyph seem to suggest, even if the feature is enabled through an option. For example, Debian does it: http://packages.debian.org/experimental/python-sqlalchemy-ext On Fri, May 7, 2010 at 19:56, Kent Bower wrote: > Just because there are configuration problems associated with adding a > feature like the one I needed is absolutely no reason to abandon it when it > can bring value to the tool if used correctly and in some circumstances. ?I > considered some of those exact complications "what if it was already > installed, etc" and with my company's project, where I am using this useful > tool in a circumstance you may overlook (it is perfectly acceptable to have > such a feature, *despite* the list of complications you mention), such a > feature would have been very valuable. > > Since it is useful in my case, I understand it would be valuable for others > as well. > > (I don't appreciate the aggressive tone of your reply, though, nor do I see > how my good faith efforts to help others warrant this... how did I possibly > offend you with my post??) > > On the other hand, I appreciate your "correct solution" as a good approach > and I'll forward this idea to an author of SQLAlchemy for his consideration. > > > On 5/7/2010 1:33 PM, Glyph Lefkowitz wrote: >> >> On May 7, 2010, at 9:09 AM, Kent wrote: >> >> >>> >>> Consider the case where you want your setup to install third-party >>> software, but you want/need to pass an argument to the command line >>> that runs "python setup.py --argument install" or "python setup.py -- >>> argument bdist_egg" >>> >>> As far as I could research, this feature is unavailable. >>> >> >> And for good reason, I should think. ?I really hope that nobody ever adds >> this feature. >> >> If you require SQLAlchemy to be installed "--with-cextensions", then what >> happens when your package is installed in an environment that already has >> SQLAlchemy installed *without* that flag? ?Does it stomp on the existing >> installation? ?What if the user installed one already with that flag and >> some other flags as well? ?What if it's system-installed and you're doing a >> user-install? >> >> Basically, compile-time and install-time options are a configuration >> nightmare. ?They should represent only how and where a package is installed, >> not what features it has. The correct solution to your problem would be to >> get SQLAlchemy to fix its broken deployment setup and split itself into 2 >> packages, "SQLAlchemyCExtensions" and "SQLAlchemy", and then have your >> project depend on both, not to try to cram installer options into the >> dependency language. >> >> For confirmation of this theory, you need look no further than the >> excruciating user-experience of source-based installation systems with >> 'variant' support, like gentoo's Portage and *BSD's Ports, versus the >> relatively non-excruciating experience of packaging systems which express >> compile-time options as different packages like Yum and Apt. -- Ga?tan de Menten From pje at telecommunity.com Fri May 7 21:12:54 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 07 May 2010 15:12:54 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <4BE44FA7.20505@retailarchitects.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> <20100507165758.ED2A83A4070@sparrow.telecommunity.com> <4BE44FA7.20505@retailarchitects.com> Message-ID: <20100507191254.89EF53A4070@sparrow.telecommunity.com> At 01:36 PM 5/7/2010 -0400, Kent Bower wrote: >>>Like I mentioned, I do have a workaround in place that is working for >>>me. I post it here in case it may help others: >> >>Please don't do that, and *nobody else should do it either*. You >>are going to ERASE people's personal distutils configuration files >>with that code! (You're not even checking to see if it exists first!) > >Thank you for posting the disclaimer before use. I am exactly aware >it is deleting that file and for my project that is completely >acceptable, so I did not do anything fancy like backing it up. I'd >hope anyone using this would have an understanding of python enough >to see what is going on and modify it to meet their needs, but that >is a poor assumption so a disclaimer would have been better; thank >you for posting such. It's not the part where it deletes *their* file that I'm worried about -- it's the people who might post such a thing to PyPI and delete random strangers' configuration files, in response to running, say, "setup.py --help". (Which your code will do.) >>I would be *extremely* upset if I tried to install your package and >>it erased one of my .pydistutils.cfg files, with all their command >>aliases, wikiup configs, path customizations, etc. > >(Again, for my project this is a non-issue, as our company installs >our software on our client's VMs.) Right - as long as you never distribute your project to *anyone else*. Ever. >>(Of course, when run under easy_install control, the sandbox should >>catch your setup script's antics and terminate it with an error >>without harming any files, but still... it's pure evil and the >>sort of thing that should NEVER, ever be done in a setup.py, on >>pain of banishment from PyPI.) >Hard to tell if you are saying that tongue-in-cheek, but if not, now >you're being a bit harsh. I am working around a problem that is a >deficit in the tool (as far as I can determine). In my >circumstances what I am doing is perfectly acceptable and >work-around of this nature are not "pure evil", it works and it may >help someone else with exactly my issue, though it was rather >amusing to read, just that I couldn't tell how serious you were being. I think perhaps you're misunderstanding me. What I'm saying is that anybody who knowingly uploads code like that to PyPI does indeed need to be banned until they get the concept of not unconditionally deleting users' configuration files in a setup.py. It's precisely this sort of behavior that easy_install's sandboxing facility was added to prevent. In any case, what I don't get is, if you're distributing this only to your customers' VMs, why not just do one of the following: 1. Put the necessary settings in lib/distutils/distutils.cfg (configure globally for the VM, in other words) 2. Put them in ~/.pydistutils.cfg to start with, and leave 'em there 3. Specify them explicitly on the command line (I assume you're not having the users manually run setup.py install or easy_install, are you?) Don't get me wrong; if this behavior really changed between c9 and c11, I would like to understand it myself. What's weird is, it seems to me that it should have either always worked or always not worked. Although, it's possible that the way it was "working" in c9 was actually a side-effect of a bug I know I fixed by c11, wherein --find-links were sometimes being taken at too high a precedence -- although, in that case, it would've only manifested in cases where the installed version and the --find-links version were identical. The major thing that changed about dependency resolution by c11, is that when a dependency is built, sys.path and the working_set are put back the way they were before the build occurred, in order to prevent certain kinds of potential recursive loops and "poisoning" of the build environment. However, the main install command should never have been bothered by that in the first place, so I'm still having trouble wrapping my head around what you're saying. I'll probably have to dig into the code more before I can get a clue on this one. From pje at telecommunity.com Fri May 7 21:26:24 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 07 May 2010 15:26:24 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <1c434699-3586-48d0-a8f3-70d2629e147d@k29g2000yqh.googlegro ups.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> <20100507165758.ED2A83A4070@sparrow.telecommunity.com> <4BE44FA7.20505@retailarchitects.com> <34b00a39-3aa6-4c25-8a2e-5ece9ddac82b@j33g2000yqn.googlegroups.com> <1c434699-3586-48d0-a8f3-70d2629e147d@k29g2000yqh.googlegroups.com> Message-ID: <20100507192623.BEFFA3A4070@sparrow.telecommunity.com> At 11:44 AM 5/7/2010 -0700, Kent wrote: > > > The next thing I would do is verify that the config file is > actually being read, by setting the DISTUTILS_DEBUG environment > variable to "yes", and then running "easy_install -v ." and > observing the output. > >I've pasted the output here: http://pastebin.com/1it5XPAL > >Hopefully you can make sense of the output. Yep. The issue isn't with installation-time dependencies, it's build-time dependencies -- that's why I was confused. There was a change to how build-time dependencies are handled, specifically intended to fix a different issue with Paste, PasteScript, and PasteDeploy's dependencies on each other. My guess is that if you put eggs for those three packages in your third-party directory, the problem (or at least the problem with those three) would go away. However, I'll see if there's a way for "child" easy_install runs (as used for build-time dependencies) to inherit the parent's settings for things like --find-links, --index-url, and --allow-hosts (but not options that would interfere with the child install's options. From kb at retailarchitects.com Fri May 7 21:55:01 2010 From: kb at retailarchitects.com (Kent Bower) Date: Fri, 07 May 2010 15:55:01 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <20100507191254.89EF53A4070@sparrow.telecommunity.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> <20100507165758.ED2A83A4070@sparrow.telecommunity.com> <4BE44FA7.20505@retailarchitects.com> <20100507191254.89EF53A4070@sparrow.telecommunity.com> Message-ID: <4BE47015.7010005@retailarchitects.com> On 5/7/2010 3:12 PM, P.J. Eby wrote: > > It's not the part where it deletes *their* file that I'm worried about > -- it's the people who might post such a thing to PyPI and delete > random strangers' configuration files, in response to running, say, > "setup.py --help". (Which your code will do.) > Ok, now we understand one another. This setuptools is very useful software even for cases where one has no intention of publicly releasing to PyPI. I failed to communicate this is the case for me.... > >>> I would be *extremely* upset if I tried to install your package and >>> it erased one of my .pydistutils.cfg files, with all their command >>> aliases, wikiup configs, path customizations, etc. >> >> (Again, for my project this is a non-issue, as our company installs >> our software on our client's VMs.) > > Right - as long as you never distribute your project to *anyone > else*. Ever. > > >>> (Of course, when run under easy_install control, the sandbox should >>> catch your setup script's antics and terminate it with an error >>> without harming any files, but still... it's pure evil and the sort >>> of thing that should NEVER, ever be done in a setup.py, on pain of >>> banishment from PyPI.) >> Hard to tell if you are saying that tongue-in-cheek, but if not, now >> you're being a bit harsh. I am working around a problem that is a >> deficit in the tool (as far as I can determine). In my circumstances >> what I am doing is perfectly acceptable and work-around of this >> nature are not "pure evil", it works and it may help someone else >> with exactly my issue, though it was rather amusing to read, just >> that I couldn't tell how serious you were being. > > I think perhaps you're misunderstanding me. What I'm saying is that > anybody who knowingly uploads code like that to PyPI does indeed need > to be banned until they get the concept of not unconditionally > deleting users' configuration files in a setup.py. > > It's precisely this sort of behavior that easy_install's sandboxing > facility was added to prevent. > I was misunderstanding you, in fact, and I wholeheartedly agree, that would be evil (well, I don't know about "evil", but very wrong)... we are on the same page now. > In any case, what I don't get is, if you're distributing this only to > your customers' VMs, why not just do one of the following: > > 1. Put the necessary settings in lib/distutils/distutils.cfg > (configure globally for the VM, in other words) > 2. Put them in ~/.pydistutils.cfg to start with, and leave 'em there > 3. Specify them explicitly on the command line (I assume you're not > having the users manually run setup.py install or easy_install, are you?) > #3 is exactly what I was trying to do... it doesn't work as of c11. #1 or #2 were my next ideas, but I really want to support the installation on these machines in more than one directory so that, once in production, it is trivial to revert to a previous release simply by running from the old release directory. Because of this, I cannot put an absolute path in lib/distutils/distutils.cfg or ~/.pydistutils.cfg because it changes relative to the thirdparty/ directory, which is relative to the location of setup.py. > Don't get me wrong; if this behavior really changed between c9 and > c11, I would like to understand it myself. What's weird is, it seems > to me that it should have either always worked or always not worked. > Although, it's possible that the way it was "working" in c9 was > actually a side-effect of a bug I know I fixed by c11, wherein > --find-links were sometimes being taken at too high a precedence -- > although, in that case, it would've only manifested in cases where the > installed version and the --find-links version were identical. > > The major thing that changed about dependency resolution by c11, is > that when a dependency is built, sys.path and the working_set are put > back the way they were before the build occurred, in order to prevent > certain kinds of potential recursive loops and "poisoning" of the > build environment. However, the main install command should never > have been bothered by that in the first place, so I'm still having > trouble wrapping my head around what you're saying. I'll probably > have to dig into the code more before I can get a clue on this one. > From kb at retailarchitects.com Fri May 7 22:03:11 2010 From: kb at retailarchitects.com (Kent Bower) Date: Fri, 07 May 2010 16:03:11 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <20100507192623.BEFFA3A4070@sparrow.telecommunity.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> <20100507165758.ED2A83A4070@sparrow.telecommunity.com> <4BE44FA7.20505@retailarchitects.com> <34b00a39-3aa6-4c25-8a2e-5ece9ddac82b@j33g2000yqn.googlegroups.com> <1c434699-3586-48d0-a8f3-70d2629e147d@k29g2000yqh.googlegroups.com> <20100507192623.BEFFA3A4070@sparrow.telecommunity.com> Message-ID: <4BE471FF.1040008@retailarchitects.com> On 5/7/2010 3:26 PM, P.J. Eby wrote: > At 11:44 AM 5/7/2010 -0700, Kent wrote: > >> > > The next thing I would do is verify that the config file is >> actually being read, by setting the DISTUTILS_DEBUG environment >> variable to "yes", and then running "easy_install -v ." and observing >> the output. >> >> I've pasted the output here: http://pastebin.com/1it5XPAL >> >> Hopefully you can make sense of the output. > > Yep. The issue isn't with installation-time dependencies, it's > build-time dependencies -- that's why I was confused. > > There was a change to how build-time dependencies are handled, > specifically intended to fix a different issue with Paste, > PasteScript, and PasteDeploy's dependencies on each other. > > My guess is that if you put eggs for those three packages in your > third-party directory, the problem (or at least the problem with those > three) would go away. > I do have these three .tar.gz files in my thirdparty/ directory. Do you mean I should try using the eggs instead and that would fix the problem because they won't need to be built, only installed? > However, I'll see if there's a way for "child" easy_install runs (as > used for build-time dependencies) to inherit the parent's settings for > things like --find-links, --index-url, and --allow-hosts (but not > options that would interfere with the child install's options. > A word of caution: I imagine there are use-cases where this is not the desired behavior. That is, in some cases, perhaps it is desirable that the dependencies do *not* inherit the command line and setup.cfg settings from the parent? But, so far as I can determine, that is how c9 worked. It might be a nice command line argument in its own regard to be able to disable this inheritance, so each dependency uses only its own setup.cfg settings - someone may *want* it to ignore the top-level settings. Thanks for your help again. Kent From glyph at twistedmatrix.com Fri May 7 22:19:02 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 7 May 2010 16:19:02 -0400 Subject: [Distutils] inability to pass setup.py command line arguments to dependency setups In-Reply-To: <4BE45459.7050907@retailarchitects.com> References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> <4BE45459.7050907@retailarchitects.com> Message-ID: <9D3FEBAE-B08D-4340-A87F-3F1C9CA03E3A@twistedmatrix.com> On May 7, 2010, at 1:56 PM, Kent Bower wrote: > (I don't appreciate the aggressive tone of your reply, though, nor do I see how my good faith efforts to help others warrant this... how did I possibly offend you with my post??) Let me start here to set the tone: Please note that I am not attacking _you_. I thought that this was pretty clear, but if anything I said (in either my previous message or this one) could be construed as a personal attack, then I sincerely apologize: that was not and is not my intention. If anything, I'm grateful to you for bringing this up, and giving me a clear example to express my views on this issue, so they are recorded for future askers of similar questions. I'm enthusiastically attacking your *proposed feature*, because I think it's a terrible idea that will cause lots of problems if implemented, but I don't believe you suggested it in bad faith or have otherwise malicious intent. > Just because there are configuration problems associated with adding a feature like the one I needed is absolutely no reason to abandon it when it can bring value to the tool if used correctly and in some circumstances. Here's a somewhat exaggerated example. It may be convenient for some people, in some circumstances, to attach an exposed lawnmower blade to a child's toy. Maybe they're out playing with their kids and want to mow the lawn at the same time: if the toys are also all lawnmowers, it's super convenient. And there's of course no problem if this feature is used correctly, in the right circumstances: by adults, carefully, to mow lawns. However, most toy manufacturers would agree that it is not worth the risk of taking off a child's hands or head to provide someone with that additional convenience. If you want a lawnmower, buy a lawnmower and use it to mow your lawn. Don't go welding exposed blades to every available household object that may be at hand when you're in your yard. > I considered some of those exact complications "what if it was already installed, etc" and with my company's project, where I am using this useful tool in a circumstance you may overlook (it is perfectly acceptable to have such a feature, *despite* the list of complications you mention), such a feature would have been very valuable. In your particular case, what you're trying to do is deploy an application, not express code dependencies. While I haven't used Buildout personally, my understanding is that this is the sort of thing that Buildout is intended to do: deploy a specific application with specific dependencies in a carefully-controlled environment. I have no idea if Buildout can do precisely what you want, but if not, there are plenty of other tools which do similar things: makefiles, shell scripts, fabric, capistrano... and each of these tools let you encode that consideration that you've done in a dedicated place, rather than injecting it somewhere it doesn't belong (your application logic, your setup.py). The risk here, which I am keen to avoid, is that if you provide the ability to pass command line arguments via setup.py, people will start uploading packages that do that to PyPI. Gradually, half of your dependencies will start wanting to pass conflicting sets of "helpful" options to their dependencies at installation time, and getting your library packages installed will be an even bigger mess than it is right now. For the most part, distutils needs *fewer* knobs that developers and packagers can turn; this is not one of the few places where it needs more. See all the recent work that's gone into statically expressing all dependencies in a data format, without even having a setup.py. This is a manifestation of a social problem in open source projects; rather than fix a problem correctly, there is a temptation to say "just let me pass through my options so it works for me". If you want to depend on C extensions in SQLAlchemy, there should be a proper way to do that so that it works regardless of what other packages you happen to have installed or how they got there. (See "standing up to user pressure", here: .) > On the other hand, I appreciate your "correct solution" as a good approach and I'll forward this idea to an author of SQLAlchemy for his consideration. Thanks. Good luck getting that fix in! :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Fri May 7 22:22:09 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 07 May 2010 16:22:09 -0400 Subject: [Distutils] setuptools passing arguments/settings to dependencies' dependencies In-Reply-To: <4BE471FF.1040008@retailarchitects.com> References: <717cf376-aef7-483a-87d1-641cb9b614f7@o11g2000yqj.googlegroups.com> <20100504183058.3B7DA3A407B@sparrow.telecommunity.com> <20100504235117.E9DBA3A407B@sparrow.telecommunity.com> <80bd4ac8-d071-4548-bb50-900fd9859bb7@a34g2000yqn.googlegroups.com> <20100507165758.ED2A83A4070@sparrow.telecommunity.com> <4BE44FA7.20505@retailarchitects.com> <34b00a39-3aa6-4c25-8a2e-5ece9ddac82b@j33g2000yqn.googlegroups.com> <1c434699-3586-48d0-a8f3-70d2629e147d@k29g2000yqh.googlegroups.com> <20100507192623.BEFFA3A4070@sparrow.telecommunity.com> <4BE471FF.1040008@retailarchitects.com> Message-ID: <20100507202208.080213A4070@sparrow.telecommunity.com> At 04:03 PM 5/7/2010 -0400, Kent Bower wrote: >On 5/7/2010 3:26 PM, P.J. Eby wrote: >>At 11:44 AM 5/7/2010 -0700, Kent wrote: >> >>> > > The next thing I would do is verify that the config file is >>> actually being read, by setting the DISTUTILS_DEBUG environment >>> variable to "yes", and then running "easy_install -v ." and >>> observing the output. >>> >>>I've pasted the output here: http://pastebin.com/1it5XPAL >>> >>>Hopefully you can make sense of the output. >> >>Yep. The issue isn't with installation-time dependencies, it's >>build-time dependencies -- that's why I was confused. >> >>There was a change to how build-time dependencies are handled, >>specifically intended to fix a different issue with Paste, >>PasteScript, and PasteDeploy's dependencies on each other. >> >>My guess is that if you put eggs for those three packages in your >>third-party directory, the problem (or at least the problem with >>those three) would go away. >I do have these three .tar.gz files in my thirdparty/ directory. Do >you mean I should try using the eggs instead and that would fix the >problem because they won't need to be built, only installed? Yes. Install-time dependency behavior didn't change between c9 and c11, and always should've worked. >>However, I'll see if there's a way for "child" easy_install runs >>(as used for build-time dependencies) to inherit the parent's >>settings for things like --find-links, --index-url, and >>--allow-hosts (but not options that would interfere with the child >>install's options. > >A word of caution: I imagine there are use-cases where this is not >the desired behavior. That is, in some cases, perhaps it is >desirable that the dependencies do *not* inherit the command line >and setup.cfg settings from the parent? I'm only referring to the options that control the locating of dependencies for the easy_install command, so that build-time dependencies are found from the same place as install-time dependencies, during a subordinate build step. That's a pretty narrow range of options and conditions. And the --find-links entries will be merged with the parent's. --index-url and --allow-hosts, on the other hand, should always be the parent's settings, and it's arguably a bug that they're not already being honored. (Apparently, not enough packages are using build-time dependencies for the problem to have been noticed and reported before now.) From mike_mp at zzzcomputing.com Fri May 7 23:21:44 2010 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Fri, 7 May 2010 17:21:44 -0400 Subject: [Distutils] [sqlalchemy] Re: inability to pass setup.py command line arguments to dependency setups In-Reply-To: References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> <4BE45459.7050907@retailarchitects.com> Message-ID: <91B97409-9775-4A70-8450-F646D5EA68E5@zzzcomputing.com> The build option for C extension or not is taken directly from the setup.py of Genshi, the template language, and as I said was also inspired by what Storm does in this regard. I think once we put the flag on by default there really won't be any controversy anymore - both of those packages build the C extension by default. It seems awkward that a source distribution (emphasis on the word "source") would be broken up into two among configuration options (not to mention it would be totally inconvenient on my side). What if our build had six flags, not two - would I then provide SQLAlchemy as six separate downloads on Pypi ? e.g. the source distro of the Python Imaging Library has a whole set of flags you can set within setup.py to assist in its location of libraries - there's not much getting around those at the level of the source distro - a source distribution only represents the source code, not its particular configuration in a particular environment. You may say these are all broken configurations, but the simple fact is that we talking about a source distribution, not a dpkg or rpm. When you get PIL as a dpkg on your Ubuntu system, its pre-configured for the correct locations of libjpeg and such. Its a problem that is solved by those packaging libraries but is not solved by Python source distros on Pypi. The setuptools analogue to dpkg and rpm, I think, would be an .egg file. If someone wants to distrbute SQLAlchemy in some kind of package oriented system with pre-selected options, outside the realm of well known unix package managers, they can always package it as such. On May 7, 2010, at 2:58 PM, Gaetan de Menten wrote: > FWIW, it is perfectly possible to package the thing separately as > Glyph seem to suggest, even if the feature is enabled through an > option. For example, Debian does it: > http://packages.debian.org/experimental/python-sqlalchemy-ext > > On Fri, May 7, 2010 at 19:56, Kent Bower wrote: >> Just because there are configuration problems associated with adding a >> feature like the one I needed is absolutely no reason to abandon it when it >> can bring value to the tool if used correctly and in some circumstances. I >> considered some of those exact complications "what if it was already >> installed, etc" and with my company's project, where I am using this useful >> tool in a circumstance you may overlook (it is perfectly acceptable to have >> such a feature, *despite* the list of complications you mention), such a >> feature would have been very valuable. >> >> Since it is useful in my case, I understand it would be valuable for others >> as well. >> >> (I don't appreciate the aggressive tone of your reply, though, nor do I see >> how my good faith efforts to help others warrant this... how did I possibly >> offend you with my post??) >> >> On the other hand, I appreciate your "correct solution" as a good approach >> and I'll forward this idea to an author of SQLAlchemy for his consideration. >> >> >> On 5/7/2010 1:33 PM, Glyph Lefkowitz wrote: >>> >>> On May 7, 2010, at 9:09 AM, Kent wrote: >>> >>> >>>> >>>> Consider the case where you want your setup to install third-party >>>> software, but you want/need to pass an argument to the command line >>>> that runs "python setup.py --argument install" or "python setup.py -- >>>> argument bdist_egg" >>>> >>>> As far as I could research, this feature is unavailable. >>>> >>> >>> And for good reason, I should think. I really hope that nobody ever adds >>> this feature. >>> >>> If you require SQLAlchemy to be installed "--with-cextensions", then what >>> happens when your package is installed in an environment that already has >>> SQLAlchemy installed *without* that flag? Does it stomp on the existing >>> installation? What if the user installed one already with that flag and >>> some other flags as well? What if it's system-installed and you're doing a >>> user-install? >>> >>> Basically, compile-time and install-time options are a configuration >>> nightmare. They should represent only how and where a package is installed, >>> not what features it has. The correct solution to your problem would be to >>> get SQLAlchemy to fix its broken deployment setup and split itself into 2 >>> packages, "SQLAlchemyCExtensions" and "SQLAlchemy", and then have your >>> project depend on both, not to try to cram installer options into the >>> dependency language. >>> >>> For confirmation of this theory, you need look no further than the >>> excruciating user-experience of source-based installation systems with >>> 'variant' support, like gentoo's Portage and *BSD's Ports, versus the >>> relatively non-excruciating experience of packaging systems which express >>> compile-time options as different packages like Yum and Apt. > > -- > Ga?tan de Menten > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From glyph at twistedmatrix.com Sat May 8 07:48:46 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sat, 8 May 2010 01:48:46 -0400 Subject: [Distutils] [sqlalchemy] Re: inability to pass setup.py command line arguments to dependency setups In-Reply-To: <50C9AE5B-63F5-4A41-B63A-A16DC654D106@zzzcomputing.com> References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> <4BE45459.7050907@retailarchitects.com> <50C9AE5B-63F5-4A41-B63A-A16DC654D106@zzzcomputing.com> Message-ID: <902B9D0F-32D8-4368-90E3-9920C806A7F6@twistedmatrix.com> On May 7, 2010, at 2:08 PM, Michael Bayer wrote: > I'd only mention that Storm has a C extension/non C extension flag as well, and only offers one source distribution on Pypi. You have to modify a variable directly within setup.py. Our setup.py features the same capability (its just our C extension is off by default for 0.6 since it was just written, which is the same case for when Storm first introduced its C extension). It occurs to me that Twisted has a similar problem (except there's no installation flag: it just builds the C extensions if it possibly can). The problem I see here is that the dependencies list of a particular project should be a complete expression of the features that the source code requires to function properly. If a C extension is present for optimization purposes only, then I don't think it ever needs to be mentioned in a dependency listing. Performance tuning is a build and deployment issue, not a dependency-correctness issue. However, if a C extension wraps features necessary for an application to work correctly, without which it will simply traceback and die, then it should be possible for the application to say "I depend on this functionality". After all, it's kind of bogus if I say "I depend on library X", and then library X gets installed, but half of it is missing for some reason. It's bogus if it's missing for any reason at all, really. The "C extension couldn't be compiled" one is common, but there are other configuration and build issues which could prevent a distribution from being fully functional. Is there already a good way to express a dependency on a portion of a source distribution, or "optional" features? A way to list one source distribution on PyPI so that it will be present under multiple names, one for each optional chunk? -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Sat May 8 10:05:39 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 8 May 2010 10:05:39 +0200 Subject: [Distutils] [sqlalchemy] Re: inability to pass setup.py command line arguments to dependency setups In-Reply-To: <902B9D0F-32D8-4368-90E3-9920C806A7F6@twistedmatrix.com> References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> <4BE45459.7050907@retailarchitects.com> <50C9AE5B-63F5-4A41-B63A-A16DC654D106@zzzcomputing.com> <902B9D0F-32D8-4368-90E3-9920C806A7F6@twistedmatrix.com> Message-ID: On Sat, May 8, 2010 at 07:48, Glyph Lefkowitz wrote: > a dependency-correctness issue. ?However, if a C extension wraps features > necessary for an application to work correctly, without which it will simply > traceback and die, then it should be possible for the application to say "I > depend on this functionality". I think setuptools can do this, but I've never used it so I'm not sure. Another option would to package the extensions as a separate package, so you get 'MySQL' and 'MySQL-C-extensions'. Problem solved. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From techtonik at gmail.com Sat May 8 14:03:46 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 8 May 2010 15:03:46 +0300 Subject: [Distutils] developing & managing multiple packages with dependencies In-Reply-To: References: Message-ID: On Fri, May 7, 2010 at 5:42 AM, Brad Allen wrote: > > Well, I am looking for stories here from other organizations as a > source of lessons. However, maybe it would help if I spent more time > describing what we're trying to accomplish. > > We're a medium size software product company (over 100 employees), the > supported platforms are Linux and Windows servers, and most of our > package dependencies are internal. However, we also have third party > open source dependencies. > >> For what is is worth keep a central repository for dependencies (either external or >> internally produced) and enforce it with an iron fist or thinks will go pretty >> quickly out of control (project A depends on project B and project C on project B' but >> projectB and projectB' cannot be used at the same time). > > With a DVCS it makes sense to have multiple repositories (a repo for > each package, er, I mean 'module distribution'), though we do have a > centralized workflow with a central server containing all the repos. You may be interested to take a look how Chrome manages it's dependencies with gclient - "A script for managing a workspace with modular dependencies that are each checked out independently from different repositories". http://dev.chromium.org/developers/how-tos/depottools A little bit tweaked it can be used to track newer commits in source repositories to give you a hint when a checkpoint in your config file can be moved forward. Starting with this you may further mark packages as autoupdating etc. -- anatoly t. From setuptools at bugs.python.org Sat May 8 14:23:04 2010 From: setuptools at bugs.python.org (techtonik) Date: Sat, 08 May 2010 12:23:04 +0000 Subject: [Distutils] [issue108] develop install fails if path to setup.py is relative In-Reply-To: <1273321384.82.0.91766926249.issue108@psf.upfronthosting.co.za> Message-ID: <1273321384.82.0.91766926249.issue108@psf.upfronthosting.co.za> New submission from techtonik : (stable011) M:\p\python\trac\env\stable011>python ..\..\bitten\bitten-0.6.x\setu p.py develop -mxd testenv\plugins running develop running egg_info writing Bitten.egg-info\PKG-INFO writing top-level names to Bitten.egg-info\top_level.txt writing dependency_links to Bitten.egg-info\dependency_links.txt writing entry points to Bitten.egg-info\entry_points.txt warning: manifest_maker: standard file 'setup.py' not found error: package directory 'bitten' does not exist (stable011) M:\p\python\trac\env\stable011>dir ..\..\bitten\bitten-0.6.x Volume in drive M is MAFN Volume Serial Number is 8EDD-9217 Directory of M:\p\python\trac\bitten\bitten-0.6.x 05/07/2010 12:35 PM . 05/07/2010 01:35 AM .. 05/06/2010 10:06 PM 524 link.PHPUnit.XML.format.diff 05/07/2010 01:36 AM 6,292 setup.py 05/07/2010 01:39 AM bitten 05/07/2010 01:36 AM 499 MANIFEST-SLAVE.in 05/07/2010 01:11 PM 2,723 rest 02/26/2010 08:04 PM 1,695 COPYING 05/07/2010 12:36 PM 71 rst2trac_and_back 05/07/2010 01:36 AM 172 setup.cfg 05/07/2010 01:36 AM 4,793 ChangeLog 02/26/2010 08:04 PM 4,990 README.txt 05/07/2010 01:37 AM Bitten.egg-info 05/07/2010 01:36 AM doc 02/26/2010 08:04 PM 374 MANIFEST.in 10 File(s) 22,133 bytes 5 Dir(s) 1,042,239,488 bytes free (stable011) M:\p\python\trac\env\stable011> ---------- messages: 517 nosy: techtonik priority: bug status: unread title: develop install fails if path to setup.py is relative _______________________________________________ Setuptools tracker _______________________________________________ From lazyweb at gmail.com Sat May 8 09:16:11 2010 From: lazyweb at gmail.com (kevin) Date: Sat, 8 May 2010 00:16:11 -0700 (PDT) Subject: [Distutils] Does buildout have a way to issue its version of sys.path? Message-ID: <5d86aba3-cb3e-427a-b5ef-d3521cb5aae2@v29g2000prb.googlegroups.com> Ok, I use vim in the console. I have python compiled into it, the system python. If i could get the value of sys.path in my buildout, I could add it to vim and then have omni completion for everything in the buildout. That's the goal. So: is there a way to get sys.path's value in a buildout? I'd like to make things a bit more automatic than copying the sys.path invocation in bin/buildout itself... From setuptools at bugs.python.org Sat May 8 15:11:04 2010 From: setuptools at bugs.python.org (techtonik) Date: Sat, 08 May 2010 13:11:04 +0000 Subject: [Distutils] [issue109] [PATCH] wrap console script into __main__ block In-Reply-To: <1273324264.52.0.532192741496.issue109@psf.upfronthosting.co.za> Message-ID: <1273324264.52.0.532192741496.issue109@psf.upfronthosting.co.za> New submission from techtonik : I would like to propose this wrapping sys.exit() calls in generated console scripts into __main__ block. This will prevent programs that inspect available modules using 'import' from exiting prematurely. See http://bitten.edgewall.org/ticket/583 Patch attached. ---------- files: wrap.scripts.into.main.diff messages: 518 nosy: techtonik priority: bug status: unread title: [PATCH] wrap console script into __main__ block Added file: http://bugs.python.org/setuptools/file68/wrap.scripts.into.main.diff _______________________________________________ Setuptools tracker _______________________________________________ -------------- next part -------------- A non-text attachment was scrubbed... Name: wrap.scripts.into.main.diff Type: application/octet-stream Size: 1309 bytes Desc: not available URL: From jim at zope.com Sat May 8 15:51:59 2010 From: jim at zope.com (Jim Fulton) Date: Sat, 8 May 2010 09:51:59 -0400 Subject: [Distutils] Does buildout have a way to issue its version of sys.path? In-Reply-To: <5d86aba3-cb3e-427a-b5ef-d3521cb5aae2@v29g2000prb.googlegroups.com> References: <5d86aba3-cb3e-427a-b5ef-d3521cb5aae2@v29g2000prb.googlegroups.com> Message-ID: On Sat, May 8, 2010 at 3:16 AM, kevin wrote: > Ok, I use vim in the console. ?I have python compiled into it, the > system python. If i could get the value of sys.path in my buildout, I > could add it ?to vim and then have omni completion for everything in > the buildout. ?That's the goal. > > So: ?is there a way to get sys.path's value in a buildout? ?I'd like > to make things a bit more automatic than copying the sys.path > invocation in bin/buildout itself... I'm not sure what you mean. Do you mean getting it as a value you could use in a variable substitution? Is so, you could write a recipe that made it available that way. Jim -- Jim Fulton From kiorky at cryptelium.net Sat May 8 16:20:10 2010 From: kiorky at cryptelium.net (kiorky) Date: Sat, 08 May 2010 16:20:10 +0200 Subject: [Distutils] Does buildout have a way to issue its version of sys.path? In-Reply-To: <5d86aba3-cb3e-427a-b5ef-d3521cb5aae2@v29g2000prb.googlegroups.com> References: <5d86aba3-cb3e-427a-b5ef-d3521cb5aae2@v29g2000prb.googlegroups.com> Message-ID: <4BE5731A.4010204@cryptelium.net> with minitage.recipe.scripts, you can generate a '.env' file, then when you source it put the appropriate 'PYTHONPATH' in the current environment. see: http://pypi.python.org/pypi/minitage.recipe.scripts/1.53#generating-an-envrionment-file I use it mainly for vim with that plugin: http://hg.cryptelium.net/hg/system/config/vim/file/6e74640f22b9/ftplugin/python/autotagsgen.vim Le 08/05/2010 09:16, kevin a ?crit : > Ok, I use vim in the console. I have python compiled into it, the > system python. If i could get the value of sys.path in my buildout, I > could add it to vim and then have omni completion for everything in > the buildout. That's the goal. > > So: is there a way to get sys.path's value in a buildout? I'd like > to make things a bit more automatic than copying the sys.path > invocation in bin/buildout itself... > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From lazyweb at gmail.com Sat May 8 19:26:52 2010 From: lazyweb at gmail.com (kevin) Date: Sat, 8 May 2010 10:26:52 -0700 (PDT) Subject: [Distutils] Does buildout have a way to issue its version of sys.path? In-Reply-To: <4BE5731A.4010204@cryptelium.net> References: <5d86aba3-cb3e-427a-b5ef-d3521cb5aae2@v29g2000prb.googlegroups.com> <4BE5731A.4010204@cryptelium.net> Message-ID: <290f8d99-82f4-4754-8ba3-16196269deba@a27g2000prj.googlegroups.com> > see:http://pypi.python.org/pypi/minitage.recipe.scripts/1.53#generating-a... That is exactly what I was looking for. Thanks! > I use it mainly for vim with that plugin:http://hg.cryptelium.net/hg/system/config/vim/file/6e74640f22b9/ftplu... Lagniappe! I did not know about this plugin. It's almost sad how happy I am about this... Much appreciated! From pje at telecommunity.com Sat May 8 21:26:37 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 08 May 2010 15:26:37 -0400 Subject: [Distutils] [issue108] develop install fails if path to setup.py is relative In-Reply-To: <1273321384.82.0.91766926249.issue108@psf.upfronthosting.co .za> References: <1273321384.82.0.91766926249.issue108@psf.upfronthosting.co.za> <1273321384.82.0.91766926249.issue108@psf.upfronthosting.co.za> Message-ID: <20100508192643.46B4A3A407B@sparrow.telecommunity.com> At 12:23 PM 5/8/2010 +0000, techtonik wrote: >New submission from techtonik : > >(stable011) M:\p\python\trac\env\stable011>python >..\..\bitten\bitten-0.6.x\setup.py develop -mxd testenv\plugins Just FYI: you can NEVER run a setup.py in a different directory than the one where you're located, whether it's with distutils or setuptools. I have no idea whether it works with distutils2 or is intended to be supported, but if it's based on disutils code, then you MUST change to the correct directory before running it. From techtonik at gmail.com Sun May 9 14:21:55 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 9 May 2010 15:21:55 +0300 Subject: [Distutils] ..\setup.py relative invocation (was: [issue108] develop install fails if path to setup.py is relative) Message-ID: setup.py fails miserably when invoked using relative path, i.e. `python ../../setup.py ...`. In this case it attempts to look up referenced files in current directory - not in its own dir. What is it needed for? Does distutils2 fix this? I ask, because it adds another "save cwd;cd;...;restore cwd" yak shaving wrapper to development process, and the error message for failure is not very intuitive. -- anatoly t. On Sat, May 8, 2010 at 10:26 PM, P.J. Eby wrote: > At 12:23 PM 5/8/2010 +0000, techtonik wrote: > >> New submission from techtonik : >> >> (stable011) M:\p\python\trac\env\stable011>python >> ..\..\bitten\bitten-0.6.x\setup.py develop -mxd testenv\plugins > > Just FYI: you can NEVER run a setup.py in a different directory than the one > where you're located, whether it's with distutils or setuptools. > > I have no idea whether it works with distutils2 or is intended to be > supported, but if it's based on disutils code, then you MUST change to the > correct directory before running it. > > From pje at telecommunity.com Sun May 9 19:28:15 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 09 May 2010 13:28:15 -0400 Subject: [Distutils] ..\setup.py relative invocation (was: [issue108] develop install fails if path to setup.py is relative) In-Reply-To: References: Message-ID: <20100509172818.C7A9E3A40A5@sparrow.telecommunity.com> At 03:21 PM 5/9/2010 +0300, anatoly techtonik wrote: >setup.py fails miserably when invoked using relative path, i.e. >`python ../../setup.py ...`. In this case it attempts to look up >referenced files in current directory - not in its own dir. > >What is it needed for? The distutils were simply written with the assumption that that's how things are... and there were never any tests that did otherwise. As a result, there's lots of code that simply operates on relative paths, beginning from the relative paths that are provided by setup(). >Does distutils2 fix this? > >I ask, because it adds another "save cwd;cd;...;restore cwd" yak >shaving wrapper to development process, and the error message for >failure is not very intuitive. FWIW, you can always use "easy_install path/to/directory", if you're just going to be running an install. I don't know if pip will wrap in this way, but it seems to me there was also a buildutils or some such package floating around a few years ago that included a simple command for running setup scripts with different interpreters, in any directory, etc. From merwok at netwok.org Sun May 9 19:52:49 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 09 May 2010 19:52:49 +0200 Subject: [Distutils] ..\setup.py relative invocation In-Reply-To: <20100509172818.C7A9E3A40A5@sparrow.telecommunity.com> References: <20100509172818.C7A9E3A40A5@sparrow.telecommunity.com> Message-ID: <4BE6F671.3000401@netwok.org> >> Does distutils2 fix this? >> >> I ask, because it adds another "save cwd;cd;...;restore cwd" yak >> shaving wrapper to development process, and the error message for >> failure is not very intuitive. You could also use a subshell (i.e. ?(cd somewhere && ./setup.py cmd)?) but that?s also an ugly workaround. make has a -C option that takes as argument the directory to cd to before running. Would something like that be ok? Regards From Nikolaus at rath.org Sun May 9 21:30:18 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sun, 09 May 2010 15:30:18 -0400 Subject: [Distutils] distribute and install --root Message-ID: <4BE70D4A.7070809@rath.org> Hello, How can I convince distribute to install a module in a different root directory, without also switching to the ~/.local/ scheme? I was surprised to find that the following doesn't work: $ python setup.py install --single-version-externally-managed --root staging/ running install running build running build_py creating build creating build/lib.linux-i686-2.6 creating build/lib.linux-i686-2.6/mymodule copying mymodule/mymodule.py -> build/lib.linux-i686-2.6/mymodule copying mymodule/__init__.py -> build/lib.linux-i686-2.6/mymodule running install_lib creating staging creating staging/home creating staging/home/nikratio creating staging/home/nikratio/.local creating staging/home/nikratio/.local/lib creating staging/home/nikratio/.local/lib/python2.6 creating staging/home/nikratio/.local/lib/python2.6/site-packages creating staging/home/nikratio/.local/lib/python2.6/site-packages/mymodule copying build/lib.linux-i686-2.6/mymodule/mymodule.py -> staging/home/nikratio/.local/lib/python2.6/site-packages/mymodule copying build/lib.linux-i686-2.6/mymodule/__init__.py -> staging/home/nikratio/.local/lib/python2.6/site-packages/mymodule [...] I want mymodule to end up somewhere in staging/usr/lib/*. Thanks, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From ziade.tarek at gmail.com Sun May 9 23:47:24 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 9 May 2010 23:47:24 +0200 Subject: [Distutils] distribute and install --root In-Reply-To: <4BE70D4A.7070809@rath.org> References: <4BE70D4A.7070809@rath.org> Message-ID: On Sun, May 9, 2010 at 9:30 PM, Nikolaus Rath wrote: > Hello, > > How can I convince distribute to install a module in a different root directory, without also switching to the ~/.local/ scheme? > > I was surprised to find that the following doesn't work: > > $ python setup.py install --single-version-externally-managed --root staging/ ... > [...] > > > I want mymodule to end up somewhere in staging/usr/lib/*. Having .local here looks like a bug to me. Could you create an issue with details ? - Name of the package you are installing (in case the setup.py has something special) - version of distribute - your OS Thanks ! Tarek From ziade.tarek at gmail.com Sun May 9 23:58:57 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 9 May 2010 23:58:57 +0200 Subject: [Distutils] ..\setup.py relative invocation (was: [issue108] develop install fails if path to setup.py is relative) In-Reply-To: References: Message-ID: On Sun, May 9, 2010 at 2:21 PM, anatoly techtonik wrote: > setup.py fails miserably when invoked using relative path, i.e. > `python ../../setup.py ...`. In this case it attempts to look up > referenced files in current directory - not in its own dir. > > What is it needed for? > Does distutils2 fix this? No, but one long term goal of distutils2 is to make setup.py optional, meaning that the commands will be run using for example a -m call that could look like this: $ python -m distutils2 run install And could take the project root path as an option : $ python -m distutils2 run install -p /here/is/the/project Regards Tarek -- Tarek Ziad? | http://ziade.org From glyph at twistedmatrix.com Mon May 10 01:41:46 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 9 May 2010 19:41:46 -0400 Subject: [Distutils] distribute and install --root In-Reply-To: References: <4BE70D4A.7070809@rath.org> Message-ID: <9167A013-7644-423C-A4EB-E10CDC453162@twistedmatrix.com> On May 9, 2010, at 5:47 PM, Tarek Ziad? wrote: > Having .local here looks like a bug to me. Could you create an issue > with details ? For what it's worth, I have had similar things happen when I had forogtten that I created a ~/.pydistutils.cfg. Have you made sure that there isn't one there? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Mon May 10 11:00:55 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 10 May 2010 11:00:55 +0200 Subject: [Distutils] distribute and install --root In-Reply-To: <9167A013-7644-423C-A4EB-E10CDC453162@twistedmatrix.com> References: <4BE70D4A.7070809@rath.org> <9167A013-7644-423C-A4EB-E10CDC453162@twistedmatrix.com> Message-ID: On Mon, May 10, 2010 at 1:41 AM, Glyph Lefkowitz wrote: > > On May 9, 2010, at 5:47 PM, Tarek Ziad? wrote: > > Having .local here looks like a bug to me. Could you create an issue > with details ? > > For what it's worth, I have had similar things happen when I had forogtten > that I created a ?~/.pydistutils.cfg. ?Have you made sure that there isn't > one there? Great reflex, thanks :) -- Tarek Ziad? | http://ziade.org From chris at simplistix.co.uk Mon May 10 14:44:07 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 10 May 2010 13:44:07 +0100 Subject: [Distutils] easy_install (and hence virtual_env, buildout, etc) versus .msi Message-ID: <4BE7FF97.4020607@simplistix.co.uk> Hi All, Is there any way to get easy_install to use .msi's as a source for distros on Windows? Twisted is only distributed as a .msi :-/ Chris From fdrake at acm.org Mon May 10 15:14:55 2010 From: fdrake at acm.org (Fred Drake) Date: Mon, 10 May 2010 09:14:55 -0400 Subject: [Distutils] ..\setup.py relative invocation In-Reply-To: <4BE6F671.3000401@netwok.org> References: <20100509172818.C7A9E3A40A5@sparrow.telecommunity.com> <4BE6F671.3000401@netwok.org> Message-ID: On Sun, May 9, 2010 at 1:52 PM, ?ric Araujo wrote: > make has a -C option that takes as argument the directory to cd to > before running. Would something like that be ok? As long as there's a setup.py, cd'ing to that directory seems better than requiring more of the end user. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From Nikolaus at rath.org Mon May 10 15:14:10 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Mon, 10 May 2010 09:14:10 -0400 Subject: [Distutils] distribute and install --root References: <4BE70D4A.7070809@rath.org> <9167A013-7644-423C-A4EB-E10CDC453162@twistedmatrix.com> Message-ID: <87bpcn99wd.fsf@inspiron.ap.columbia.edu> Glyph Lefkowitz writes: > On May 9, 2010, at 5:47 PM, Tarek Ziad? wrote: > >> Having .local here looks like a bug to me. Could you create an issue >> with details ? > > For what it's worth, I have had similar things happen when I had > forogtten that I created a ~/.pydistutils.cfg. Have you made sure that > there isn't one there? No, that was it. And I certainly don't remember ever touching or creating that file either. Thanks! Best, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From glyph at twistedmatrix.com Mon May 10 20:30:13 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 10 May 2010 14:30:13 -0400 Subject: [Distutils] distribute and install --root In-Reply-To: <87bpcn99wd.fsf@inspiron.ap.columbia.edu> References: <4BE70D4A.7070809@rath.org> <9167A013-7644-423C-A4EB-E10CDC453162@twistedmatrix.com> <87bpcn99wd.fsf@inspiron.ap.columbia.edu> Message-ID: On May 10, 2010, at 9:14 AM, Nikolaus Rath wrote: > Glyph Lefkowitz writes: >> On May 9, 2010, at 5:47 PM, Tarek Ziad? wrote: >> >>> Having .local here looks like a bug to me. Could you create an issue >>> with details ? >> >> For what it's worth, I have had similar things happen when I had >> forogtten that I created a ~/.pydistutils.cfg. Have you made sure that >> there isn't one there? > > No, that was it. And I certainly don't remember ever touching or > creating that file either. Thanks! Does distutils have a flag for ignoring ~/.pydistutils.cfg, by the way? I hit this fairly often and it's always a nasty surprise *right* at the end of a build process. It would be nice to have all official-release scripts specify an option that specifically overrides per-user configuration. From tseaver at palladion.com Mon May 10 20:35:11 2010 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 10 May 2010 14:35:11 -0400 Subject: [Distutils] [sqlalchemy] Re: inability to pass setup.py command line arguments to dependency setups In-Reply-To: References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> <4BE45459.7050907@retailarchitects.com> <50C9AE5B-63F5-4A41-B63A-A16DC654D106@zzzcomputing.com> <902B9D0F-32D8-4368-90E3-9920C806A7F6@twistedmatrix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lennart Regebro wrote: > On Sat, May 8, 2010 at 07:48, Glyph Lefkowitz wrote: >> a dependency-correctness issue. However, if a C extension wraps features >> necessary for an application to work correctly, without which it will simply >> traceback and die, then it should be possible for the application to say "I >> depend on this functionality". > > I think setuptools can do this, but I've never used it so I'm not sure. > > Another option would to package the extensions as a separate package, > so you get 'MySQL' and 'MySQL-C-extensions'. Problem solved. :) That is effectivaly what the "extras" bit in setuptools does[ http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvoUd8ACgkQ+gerLs4ltQ6zxQCePHJfWk8mvoQrt7k8n8s4TfOk FmAAnAl3XT1HZK1oeP2Yq1HgLzQQJHTf =HZXH -----END PGP SIGNATURE----- From regebro at gmail.com Mon May 10 21:20:41 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 10 May 2010 21:20:41 +0200 Subject: [Distutils] [sqlalchemy] Re: inability to pass setup.py command line arguments to dependency setups In-Reply-To: References: <9E65CBC1-F3AB-46FD-B89F-208F801887DE@twistedmatrix.com> <4BE45459.7050907@retailarchitects.com> <50C9AE5B-63F5-4A41-B63A-A16DC654D106@zzzcomputing.com> <902B9D0F-32D8-4368-90E3-9920C806A7F6@twistedmatrix.com> Message-ID: On Mon, May 10, 2010 at 20:35, Tres Seaver wrote: > http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies Right, that's what I was thinking of. Clearly in this case, the MySQL package should declare it's C-extensions as an "extra". -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From matt at tplus1.com Mon May 10 21:38:33 2010 From: matt at tplus1.com (Matthew Wilson) Date: Mon, 10 May 2010 19:38:33 +0000 (UTC) Subject: [Distutils] How to track performance improvements in my code? Message-ID: I know how to use timeit and/or profile to measure the current run-time cost of some code. I want to record the time used by some original implementation, then after I rewrite it, I want to find out if I made stuff faster or slower, and by how much. Other than me writing down numbers on a piece of paper on my desk, does some tool that does this already exist? If it doesn't exist, how should I build it? Matt From ziade.tarek at gmail.com Mon May 10 22:05:44 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 10 May 2010 22:05:44 +0200 Subject: [Distutils] distribute and install --root In-Reply-To: References: <4BE70D4A.7070809@rath.org> <9167A013-7644-423C-A4EB-E10CDC453162@twistedmatrix.com> <87bpcn99wd.fsf@inspiron.ap.columbia.edu> Message-ID: On Mon, May 10, 2010 at 8:30 PM, Glyph Lefkowitz wrote: > > On May 10, 2010, at 9:14 AM, Nikolaus Rath wrote: > >> Glyph Lefkowitz writes: >>> On May 9, 2010, at 5:47 PM, Tarek Ziad? wrote: >>> >>>> Having .local here looks like a bug to me. Could you create an issue >>>> with details ? >>> >>> For what it's worth, I have had similar things happen when I had >>> forogtten that I created a ~/.pydistutils.cfg. Have you made sure that >>> there isn't one there? >> >> No, that was it. And I certainly don't remember ever touching or >> creating that file either. Thanks! > > Does distutils have a flag for ignoring ~/.pydistutils.cfg, by the way? ?I hit this fairly often and it's always a nasty surprise *right* at the end of a build process. ?It would be nice to have all official-release scripts specify an option that specifically overrides per-user configuration. Yes, someone provided this patch a while ago and I've added it (it's "--no-user-cfg" in 2.6). We should ask people to run this option to get a pseudo-similar execution context when we track a bug > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From regebro at gmail.com Mon May 10 22:06:06 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 10 May 2010 22:06:06 +0200 Subject: [Distutils] How to track performance improvements in my code? In-Reply-To: References: Message-ID: On Mon, May 10, 2010 at 21:38, Matthew Wilson wrote: > > I know how to use timeit and/or profile to measure the current run-time > cost of some code. > > I want to record the time used by some original implementation, then > after I rewrite it, I want to find out if I made stuff faster or slower, > and by how much. > > Other than me writing down numbers on a piece of paper on my desk, does > some tool that does this already exist? > > If it doesn't exist, how should I build it? Since circumstances change, most importantly hardware, the only way to be sure is to run the performance test on all versions on the same machine (obviously while it's not doing anything else). A system that can take a bunch of svn/hg/whatever tags, check them out build them, and test them as a a big batch could be helpful. Don't really see what it has to do with distutils... -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ziade.tarek at gmail.com Mon May 10 22:11:10 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 10 May 2010 22:11:10 +0200 Subject: [Distutils] How to track performance improvements in my code? In-Reply-To: References: Message-ID: On Mon, May 10, 2010 at 10:06 PM, Lennart Regebro wrote: > On Mon, May 10, 2010 at 21:38, Matthew Wilson wrote: >> >> I know how to use timeit and/or profile to measure the current run-time >> cost of some code. >> >> I want to record the time used by some original implementation, then >> after I rewrite it, I want to find out if I made stuff faster or slower, >> and by how much. >> >> Other than me writing down numbers on a piece of paper on my desk, does >> some tool that does this already exist? >> >> If it doesn't exist, how should I build it? > > Since circumstances change, most importantly hardware, the only way to > be sure is to run the performance test on all versions on the same > machine (obviously while it's not doing anything else). A system that > can take a bunch of svn/hg/whatever tags, check them out build them, > and test them as a a big batch could be helpful. you can also translate the results into pystones, so your output is roughly the same on every machine that has a similar architecture / OS I guess. > > Don't really see what it has to do with distutils... I think Matthew also wanted a way to record previous runs and diff them. That would be a good question for the Testing In Python mailing list, > -- > Lennart Regebro: Python, Zope, Plone, Grok > http://regebro.wordpress.com/ > +33 661 58 14 64 > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From matt at tplus1.com Mon May 10 22:11:53 2010 From: matt at tplus1.com (Matthew Wilson) Date: Mon, 10 May 2010 20:11:53 +0000 (UTC) Subject: [Distutils] How to track performance improvements in my code? References: Message-ID: On Mon 10 May 2010 04:06:06 PM EDT, Lennart Regebro wrote: > Don't really see what it has to do with distutils... Yeah, whoops! I sent to the wrong group. Well thanks for the advice! Matt From glyph at twistedmatrix.com Tue May 11 02:38:57 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 10 May 2010 20:38:57 -0400 Subject: [Distutils] distribute and install --root In-Reply-To: References: <4BE70D4A.7070809@rath.org> <9167A013-7644-423C-A4EB-E10CDC453162@twistedmatrix.com> <87bpcn99wd.fsf@inspiron.ap.columbia.edu> Message-ID: <2E267FD8-561F-44DB-BA1A-A6B2D9207280@twistedmatrix.com> On May 10, 2010, at 4:05 PM, Tarek Ziad? wrote: > Yes, someone provided this patch a while ago and I've added it (it's > "--no-user-cfg" in 2.6). Do you mean in 2.7? I see it documented here: , but not here: . -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed May 12 10:57:48 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 12 May 2010 09:57:48 +0100 Subject: [Distutils] [buildout] ${} substitutions don't happen in buildout:extends? In-Reply-To: References: <4BE28807.9090707@simplistix.co.uk> Message-ID: <4BEA6D8C.8070302@simplistix.co.uk> Jim Fulton wrote: >> For example, here's a project buildout: >> >> [buildout] >> extends = ${buildout:base_location}/versions/repoze.bfg-1.2-versions.cfg >> ... >> >> And here's what's in ~/.buildout/default.cfg: >> >> [buildout] >> base_location = /Users/chris/LocalSVN/base >> >> ...but, the normal ${} substitution doesn't appear to happen in the extends >> line, so I get the following error: >> >> IOError: [Errno 2] No such file or directory: >> '/Users/chris.withers/LocalSVN/demoapp/${buildout:base_location}/versions/repoze.bfg-1.2-versions.cfg' >> >> What am I doing wrong? > > Nothing. There are certtain stages of processing where variable > substitution isn't done. > This is, in part because the variable database hasn't been established yet. > > I think the situation needs to be improved. At least, I think variable > substitutions > could reflect the database as it exists at the time of a variables use. This > would allow extends to use variables defined on the command line, > in default.cfg and in files already read. Of course, the restrictions > that remain should be clearly documented. This all sounds great. I hope it gets implemented some time, if you need anything testing, please let me know! cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Wed May 12 14:17:45 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 12 May 2010 13:17:45 +0100 Subject: [Distutils] [buildout] extending a .cfg supplied by another package Message-ID: <4BEA9C69.8060602@simplistix.co.uk> Hi All, I'd like to be able to do: [buildout] extends=some.package:aconfig.cfg ...kinda like you can with templates in BFG, which I believe uses pkg_tools to do the hard work. What're the chances of that happening? cheers, Chris From jim at zope.com Wed May 12 21:51:19 2010 From: jim at zope.com (Jim Fulton) Date: Wed, 12 May 2010 15:51:19 -0400 Subject: [Distutils] [buildout] extending a .cfg supplied by another package In-Reply-To: <4BEA9C69.8060602@simplistix.co.uk> References: <4BEA9C69.8060602@simplistix.co.uk> Message-ID: On Wed, May 12, 2010 at 8:17 AM, Chris Withers wrote: > Hi All, > > I'd like to be able to do: > > [buildout] > extends=some.package:aconfig.cfg That's an interesting idea. > ...kinda like you can with templates in BFG, which I believe uses pkg_tools > to do the hard work. > > What're the chances of that happening? Low, in the short term. A good first step would be to work out a somewhat detailed proposal including: - Descriptions of specific situations where this would solve a problem, with specific examples of how the proposed meachanism would be applied. - A syntax proposal. It has to address distinguishing file paths from urls from these new things. I suspect these should be expressed as URLs, e.g.: project://some.project/aconfig.cfg I'll note that I suspect you can do this now using a buildout extension that installs a new URL handler in urtllib2. Jim -- Jim Fulton From ws at gocept.com Fri May 14 09:47:50 2010 From: ws at gocept.com (Wolfgang Schnerring) Date: Fri, 14 May 2010 09:47:50 +0200 Subject: [Distutils] [buildout] extending a .cfg supplied by another package References: <4BEA9C69.8060602@simplistix.co.uk> Message-ID: <87zl02c4bd.fsf@elzar.ws.whq.gocept.com> * Jim Fulton [2010-05-12 21:51]: > Chris Withers wrote: > > I'd like to be able to do: > > [buildout] > > extends=some.package:aconfig.cfg > - Descriptions of specific situations where this would solve a > problem, with specific examples of how the proposed meachanism would > be applied. I don't know what Chris had in mind, but one use case for me is versioning the buildout configuration and keeping it in sync with the package I'm buildouting. For example, say the way a package is deployed changes from its version 1.5 to 2.0, so some buildout parts need to work differently than before, e. g. because the appserver needs to be configured differently now. I'm not aware of a good way to deal with that currently, other than using an SCM tag for the buildout config files. But this is cumbersome, as a) it's an additional thing you need to do when cutting a release and b) you need to take care to first figure out and then check out the matching buildout config version for the package version you want to deploy. If I could put the relevant buildout configuration into the egg itself, that would handle this issue nicely. > - A syntax proposal. It has to address distinguishing file paths from > urls from these new things. I suspect these should be expressed as > URLs, e.g.: > project://some.project/aconfig.cfg I'd propose egg://some.dotted.name/config.cfg > I'll note that I suspect you can do this now using a buildout > extension that installs a new URL handler in urtllib2. That's an interesting approach. I'd definitely want to use buildout's egg version handling/pinning for the "extends egg", but yeah, maybe that'd be possible that way, too. Wolfgang From chris at simplistix.co.uk Fri May 14 10:56:51 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 14 May 2010 09:56:51 +0100 Subject: [Distutils] [buildout] extending a .cfg supplied by another package In-Reply-To: References: <4BEA9C69.8060602@simplistix.co.uk> Message-ID: <4BED1053.2050805@simplistix.co.uk> Jim Fulton wrote: > On Wed, May 12, 2010 at 8:17 AM, Chris Withers wrote: >> Hi All, >> >> I'd like to be able to do: >> >> [buildout] >> extends=some.package:aconfig.cfg > > That's an interesting idea. Yeah, I finally took a look at the package_resources part of setuptools ;-) > Low, in the short term. A good first step would be to work out a > somewhat detailed proposal including: > > - Descriptions of specific situations where this would solve a > problem, So, we have a bunch of packages that all depend on each other. We group all the boilerplate into one package, including things like base buildout .cfg files and the like. We currently have to specify http urls to tags on our svn server to extend these .cfg's, it would be better if we could just say "extend this configuration from that package" > with specific examples of how the proposed meachanism would > be applied. The tricky bit from my perspective is that some.package would need to be obtained before any parts had been run, but I guess the same is true of repice packages and buildout extension packages, so I'm guessing this problem has been solved already? > - A syntax proposal. It has to address distinguishing file paths from > urls from these new things. I suspect these should be expressed as > URLs, e.g.: > > project://some.project/aconfig.cfg > > I'll note that I suspect you can do this now using a buildout > extension that installs a new URL handler in urtllib2. That's an interesting idea! I'll certainly have a look... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From tseaver at palladion.com Fri May 14 16:10:20 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 14 May 2010 10:10:20 -0400 Subject: [Distutils] [buildout] extending a .cfg supplied by another package In-Reply-To: <87zl02c4bd.fsf@elzar.ws.whq.gocept.com> References: <4BEA9C69.8060602@simplistix.co.uk> <87zl02c4bd.fsf@elzar.ws.whq.gocept.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Wolfgang Schnerring wrote: > * Jim Fulton [2010-05-12 21:51]: >> Chris Withers wrote: >>> I'd like to be able to do: >>> [buildout] >>> extends=some.package:aconfig.cfg >> - Descriptions of specific situations where this would solve a >> problem, with specific examples of how the proposed meachanism would >> be applied. > > I don't know what Chris had in mind, but one use case for me is > versioning the buildout configuration and keeping it in sync with the > package I'm buildouting. For example, say the way a package is deployed > changes from its version 1.5 to 2.0, so some buildout parts need to work > differently than before, e. g. because the appserver needs to be > configured differently now. > > I'm not aware of a good way to deal with that currently, other than > using an SCM tag for the buildout config files. But this is cumbersome, > as a) it's an additional thing you need to do when cutting a release and > b) you need to take care to first figure out and then check out the > matching buildout config version for the package version you want to > deploy. > > If I could put the relevant buildout configuration into the egg itself, > that would handle this issue nicely. > >> - A syntax proposal. It has to address distinguishing file paths from >> urls from these new things. I suspect these should be expressed as >> URLs, e.g.: >> project://some.project/aconfig.cfg > > I'd propose egg://some.dotted.name/config.cfg I think egg:some.dotted.name#config.cfg would be clearer: the "hierarchical" URL form is misleading. We could even adopt the convention that such config files needed to be registered as entry points under some given section, e.g.: setup( ... entry_points = { 'zc.buildout.cfg': {'buildout.cfg': './buildout.cfg', 'develop.cfg': './develop.cfg'}, } ) perhaps with a fallback to searching in the root directory of the egg for unregistered files. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvtWcwACgkQ+gerLs4ltQ42wgCgmUfJoETb3D+Ogckby4WIVols N+wAn0I94tV91vB821sTOdtoxjDxw4M/ =0jjq -----END PGP SIGNATURE----- From manlio.perillo at gmail.com Fri May 14 17:08:44 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Fri, 14 May 2010 17:08:44 +0200 Subject: [Distutils] using sub_commands in distutils Message-ID: <4BED677C.5090804@gmail.com> Hi. In a package, I have gettext catalog messages, and I want to compile them when the package is build. I looked at the Mercurial setup.py script, and what it does is: from distutils.command.build import build build.sub_commands.insert(0, ('build_mo', None)) Is this the correct way? Thanks Manlio From pje at telecommunity.com Fri May 14 17:38:30 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 14 May 2010 11:38:30 -0400 Subject: [Distutils] using sub_commands in distutils In-Reply-To: <4BED677C.5090804@gmail.com> References: <4BED677C.5090804@gmail.com> Message-ID: <20100514153829.894C33A4061@sparrow.telecommunity.com> At 05:08 PM 5/14/2010 +0200, Manlio Perillo wrote: >Hi. > >In a package, I have gettext catalog messages, and I want to compile >them when the package is build. > >I looked at the Mercurial setup.py script, and what it does is: > >from distutils.command.build import build >build.sub_commands.insert(0, ('build_mo', None)) > > >Is this the correct way? No. The correct way is to subclass the build command and use a cmdclass dictionary in the setup() call. Something like: from distutils.command.build import build class build(build): sub_commands = [('build_mo', None)]+ build.sub_commands setup( ... cmdclass = dict(build=build) ) The way Mercurial is doing it is monkeypatching that will break if the current Python process runs more than one setup()... e.g. when the setup script is being run by easy_install or some similar tool. (As you can see, though, it only adds one line of code to do it the correct way, as documented in the distutils manual.) From manlio_perillo at libero.it Fri May 14 17:53:03 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Fri, 14 May 2010 17:53:03 +0200 Subject: [Distutils] using sub_commands in distutils In-Reply-To: <20100514153829.894C33A4061@sparrow.telecommunity.com> References: <4BED677C.5090804@gmail.com> <20100514153829.894C33A4061@sparrow.telecommunity.com> Message-ID: <4BED71DF.4050007@libero.it> P.J. Eby ha scritto: > At 05:08 PM 5/14/2010 +0200, Manlio Perillo wrote: >> Hi. >> >> In a package, I have gettext catalog messages, and I want to compile >> them when the package is build. >> >> I looked at the Mercurial setup.py script, and what it does is: >> >> from distutils.command.build import build >> build.sub_commands.insert(0, ('build_mo', None)) >> >> >> Is this the correct way? > > No. The correct way is to subclass the build command and use a cmdclass > dictionary in the setup() call. Something like: > > from distutils.command.build import build > class build(build): > sub_commands = [('build_mo', None)]+ build.sub_commands > Ok, thanks. By the way: in order to get messages compiled, should I just subclass build and develop commands? Regards Manlio From pje at telecommunity.com Fri May 14 18:03:43 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 14 May 2010 12:03:43 -0400 Subject: [Distutils] using sub_commands in distutils In-Reply-To: <4BED71DF.4050007@libero.it> References: <4BED677C.5090804@gmail.com> <20100514153829.894C33A4061@sparrow.telecommunity.com> <4BED71DF.4050007@libero.it> Message-ID: <20100514160342.A485B3A4061@sparrow.telecommunity.com> At 05:53 PM 5/14/2010 +0200, Manlio Perillo wrote: >By the way: in order to get messages compiled, should I just subclass >build and develop commands? I don't understand your question. Do you mean "should I just subclass the 'build' command and the 'develop' command"? or "should I just subclass the 'build' command and develop my own commands based on that?" or "should I just subclass the 'build' command and develop my own commands from scratch without subclassing?" Personally, however, if I had to do what you're doing, I'd package a build_mo command as a setuptools plugin, and add it to my build_requires dependencies. That way, I could use it with multiple projects. I'd still have to do the short 'build' subclass in the setup.py, but it's not a lot of code to add. From manlio_perillo at libero.it Fri May 14 18:10:42 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Fri, 14 May 2010 18:10:42 +0200 Subject: [Distutils] using sub_commands in distutils In-Reply-To: <20100514160342.A485B3A4061@sparrow.telecommunity.com> References: <4BED677C.5090804@gmail.com> <20100514153829.894C33A4061@sparrow.telecommunity.com> <4BED71DF.4050007@libero.it> <20100514160342.A485B3A4061@sparrow.telecommunity.com> Message-ID: <4BED7602.5030009@libero.it> P.J. Eby ha scritto: > At 05:53 PM 5/14/2010 +0200, Manlio Perillo wrote: >> By the way: in order to get messages compiled, should I just subclass >> build and develop commands? > > I don't understand your question. > I want messages to be compiled (using compile_catalog distutils command from babel) in all these cases: 1) create a binary distribution 2) create an egg 3) running the setup using develop command > [...] > > Personally, however, if I had to do what you're doing, I'd package a > build_mo command as a setuptools plugin, and add it to my build_requires > dependencies. That way, I could use it with multiple projects. This is indeed what I'm doing, using babel. However I noted that running python setup.py develop does not execute the compile_catalog command. > I'd > still have to do the short 'build' subclass in the setup.py, but it's > not a lot of code to add. > Thanks Manlio From pje at telecommunity.com Fri May 14 18:55:42 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 14 May 2010 12:55:42 -0400 Subject: [Distutils] using sub_commands in distutils In-Reply-To: <4BED7602.5030009@libero.it> References: <4BED677C.5090804@gmail.com> <20100514153829.894C33A4061@sparrow.telecommunity.com> <4BED71DF.4050007@libero.it> <20100514160342.A485B3A4061@sparrow.telecommunity.com> <4BED7602.5030009@libero.it> Message-ID: <20100514165540.DB8053A4061@sparrow.telecommunity.com> At 06:10 PM 5/14/2010 +0200, Manlio Perillo wrote: >P.J. Eby ha scritto: > > At 05:53 PM 5/14/2010 +0200, Manlio Perillo wrote: > >> By the way: in order to get messages compiled, should I just subclass > >> build and develop commands? > > > > I don't understand your question. > > > >I want messages to be compiled (using compile_catalog distutils command >from babel) in all these cases: >1) create a binary distribution >2) create an egg >3) running the setup using develop command > > > > [...] > > > > Personally, however, if I had to do what you're doing, I'd package a > > build_mo command as a setuptools plugin, and add it to my build_requires > > dependencies. That way, I could use it with multiple projects. > >This is indeed what I'm doing, using babel. > >However I noted that running > python setup.py develop > >does not execute the compile_catalog command. Ah. Okay, so yes, you'd need to subclass develop as well as bdist_egg. An easier way, however, might be to define aliases in setup.cfg: [alias] develop = build_mo develop bdist_egg = build_mo bdist_egg ... etc. Another alternative would be to use an egg_info.writers entry point -- the egg_info command gets called for install, develop, bdist_egg, and all other bdist_* installation scenarios. The doc is here: http://peak.telecommunity.com/DevCenter/setuptools#adding-new-egg-info-files It's a bit of a hack, in that you wouldn't actually be generating an egg-info file. You'd do something like: def my_writer(cmd, basename, filename): cmd.run_command('build_mo') as your writer function. Sadly, this is perhaps the easiest way to extend the build process in the current distribution tools environment. (Hopefully, the build_mo command will not do too much unnecessary processing.) Still another way to accomplish this, would be to add your built files to your source distribution(s), so that recipients don't need to do it themselves. (Of course, that won't work if the files are platform-specific.) From merwok at netwok.org Fri May 14 19:22:50 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 14 May 2010 19:22:50 +0200 Subject: [Distutils] using sub_commands in distutils In-Reply-To: <20100514165540.DB8053A4061@sparrow.telecommunity.com> References: <4BED677C.5090804@gmail.com> <20100514153829.894C33A4061@sparrow.telecommunity.com> <4BED71DF.4050007@libero.it> <20100514160342.A485B3A4061@sparrow.telecommunity.com> <4BED7602.5030009@libero.it> <20100514165540.DB8053A4061@sparrow.telecommunity.com> Message-ID: <4BED86EA.4060209@netwok.org> Hello For the record, I wondered whether aliases in config files were a feature of Distutils or Setuptools, and it turns out it?s Setuptools. Would that be worth inclusion in Distutils2? Regards From manlio_perillo at libero.it Fri May 14 19:22:40 2010 From: manlio_perillo at libero.it (Manlio Perillo) Date: Fri, 14 May 2010 19:22:40 +0200 Subject: [Distutils] using sub_commands in distutils In-Reply-To: <20100514165540.DB8053A4061@sparrow.telecommunity.com> References: <4BED677C.5090804@gmail.com> <20100514153829.894C33A4061@sparrow.telecommunity.com> <4BED71DF.4050007@libero.it> <20100514160342.A485B3A4061@sparrow.telecommunity.com> <4BED7602.5030009@libero.it> <20100514165540.DB8053A4061@sparrow.telecommunity.com> Message-ID: <4BED86E0.6070205@libero.it> P.J. Eby ha scritto: > [...] >> I want messages to be compiled (using compile_catalog distutils command >> from babel) in all these cases: >> 1) create a binary distribution >> 2) create an egg >> 3) running the setup using develop command >> >> >> > [...] > [...] >> However I noted that running >> python setup.py develop >> >> does not execute the compile_catalog command. > > Ah. Okay, so yes, you'd need to subclass develop as well as bdist_egg. I have subclassed develop command: class develop(develop): sub_commands = [('compile_catalog', None)] + develop.sub_commands cmdclass = dict(build=build, develop=develop), However compile_catalog is not called. > An easier way, however, might be to define aliases in setup.cfg: > > [alias] > develop = build_mo develop > bdist_egg = build_mo bdist_egg Thanks, this seems a good solution. I think I will use this. Note however that the configuration group is "aliases", not "alias". It is unfortunate that aliases are not executed internally. That is, I would really like to define an alias for the "build" command, and have it expanded when "build" command is called by other commands like "bdist". > [...] > Still another way to accomplish this, would be to add your built files > to your source distribution(s), so that recipients don't need to do it > themselves. (Of course, that won't work if the files are > platform-specific.) > In my case this is not a problem, but I prefer to generate them during setup, if this is not inconvenient. Regards Manlio From pje at telecommunity.com Fri May 14 20:27:43 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 14 May 2010 14:27:43 -0400 Subject: [Distutils] using sub_commands in distutils In-Reply-To: <4BED86E0.6070205@libero.it> References: <4BED677C.5090804@gmail.com> <20100514153829.894C33A4061@sparrow.telecommunity.com> <4BED71DF.4050007@libero.it> <20100514160342.A485B3A4061@sparrow.telecommunity.com> <4BED7602.5030009@libero.it> <20100514165540.DB8053A4061@sparrow.telecommunity.com> <4BED86E0.6070205@libero.it> Message-ID: <20100514182742.7EA113A4061@sparrow.telecommunity.com> At 07:22 PM 5/14/2010 +0200, Manlio Perillo wrote: >P.J. Eby ha scritto: > > [...] > >> I want messages to be compiled (using compile_catalog distutils command > >> from babel) in all these cases: > >> 1) create a binary distribution > >> 2) create an egg > >> 3) running the setup using develop command > >> > >> > >> > [...] > > [...] > >> However I noted that running > >> python setup.py develop > >> > >> does not execute the compile_catalog command. > > > > Ah. Okay, so yes, you'd need to subclass develop as well as bdist_egg. > >I have subclassed develop command: > >class develop(develop): > sub_commands = [('compile_catalog', None)] + develop.sub_commands > >cmdclass = dict(build=build, develop=develop), > >However compile_catalog is not called. Develop doesn't have or run any subcommands; that's specific to the 'build' command. Any command you subclass, you'll have to investigate the source of to find out what it does and the best place to hook into it. > > An easier way, however, might be to define aliases in setup.cfg: > > > > [alias] > > develop = build_mo develop > > bdist_egg = build_mo bdist_egg > >Thanks, this seems a good solution. >I think I will use this. > >Note however that the configuration group is "aliases", not "alias". Ah, yes, sorry. >It is unfortunate that aliases are not executed internally. >That is, I would really like to define an alias for the "build" command, >and have it expanded when "build" command is called by other commands >like "bdist". Yeah, that could lead to other problems in the current distutils architecture. A redesigned distutils system would hopefully have a better way to accomplish this. From sridharr at activestate.com Sat May 15 00:30:03 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Fri, 14 May 2010 15:30:03 -0700 Subject: [Distutils] Distribute 0.6.12 broken on Windows Message-ID: May I call your attention to this bug? http://bitbucket.org/tarek/distribute/issue/151/ 0.6.12 broke Windows installation. Are we not planning a new release? -srid From ziade.tarek at gmail.com Sat May 15 10:04:37 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 15 May 2010 03:04:37 -0500 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: References: Message-ID: On Fri, May 14, 2010 at 5:30 PM, Sridhar Ratnakumar wrote: > May I call your attention to this bug? > > ?http://bitbucket.org/tarek/distribute/issue/151/ > > 0.6.12 broke Windows installation. Are we not planning a new release? I was travelling, so I couldn't look at this yet. I'll be able to look at it today, we need to understand why this folder can be present and empty under some circumstances, and what it means. Notice that the whole point of Distribute is to allow packaging people to fix bugs and not depend on one person. IIRC you have a commit access so you can fix bugs too ;) And I'd be glad to give you a pypi access and to show you how to do releases Regards Tarek > > -srid > > -- Tarek Ziad? | http://ziade.org From ws at gocept.com Sat May 15 11:04:13 2010 From: ws at gocept.com (Wolfgang Schnerring) Date: Sat, 15 May 2010 11:04:13 +0200 Subject: [Distutils] [buildout] extending a .cfg supplied by another package References: <4BEA9C69.8060602@simplistix.co.uk> <87zl02c4bd.fsf@elzar.ws.whq.gocept.com> Message-ID: <87vdapbkoi.fsf@elzar.ws.whq.gocept.com> * Tres Seaver [2010-05-14 16:10]: > Wolfgang Schnerring wrote: >> * Jim Fulton [2010-05-12 21:51]: >>> - A syntax proposal. It has to address distinguishing file paths from >>> urls from these new things. I suspect these should be expressed as >>> URLs, e.g.: >>> project://some.project/aconfig.cfg >> >> I'd propose egg://some.dotted.name/config.cfg > > I think egg:some.dotted.name#config.cfg would be clearer: the > "hierarchical" URL form is misleading. Agreed, that makes sense to me. > We could even adopt the convention that such config files needed to be > registered as entry points under some given section, e.g.: Using entry points sounds good. One could even define a default one (named... "default", maybe? ;), so you could just reference egg://some.dotted.name without a #filename.cfg modifier. Wolfgang From ziade.tarek at gmail.com Sat May 15 11:31:18 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 15 May 2010 11:31:18 +0200 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: References: Message-ID: Jason, would you mind looking at your fix for the sandbox failure you have added in r670 to fix #118? It seems that this code will break if the pywin32 package is not installed as expected. Srid, is the pywon32 installed as a zipped egg in your environment ? if so, could you unzip it to see if the bug goes away ? In any case, if pywin32 uses this directory to write files in it, I don't see how this can work with zipped eggs. A quick fix is to check if we are able to read the path, and if not just drop adding this in the sandbox. On Sat, May 15, 2010 at 10:04 AM, Tarek Ziad? wrote: > On Fri, May 14, 2010 at 5:30 PM, Sridhar Ratnakumar > wrote: >> May I call your attention to this bug? >> >> ?http://bitbucket.org/tarek/distribute/issue/151/ >> >> 0.6.12 broke Windows installation. Are we not planning a new release? > > I was travelling, so I couldn't look at this yet. I'll be able to look > at it today, > we need to understand why this folder can be present and empty under some > circumstances, and what it means. > > Notice that the whole point of Distribute is to allow packaging people > to fix bugs and not > depend on one person. IIRC you have a commit access so you can fix bugs too ;) > > And I'd be glad to give you a pypi access and to show you how to do releases > > Regards > Tarek > > > >> >> -srid >> >> > > > > -- > Tarek Ziad? | http://ziade.org > -- Tarek Ziad? | http://ziade.org From chris at simplistix.co.uk Mon May 17 12:58:30 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 17 May 2010 11:58:30 +0100 Subject: [Distutils] [buildout] extending a .cfg supplied by another package In-Reply-To: References: <4BEA9C69.8060602@simplistix.co.uk> <87zl02c4bd.fsf@elzar.ws.whq.gocept.com> Message-ID: <4BF12156.5080508@simplistix.co.uk> Tres Seaver wrote: >>> project://some.project/aconfig.cfg >> I'd propose egg://some.dotted.name/config.cfg > > I think egg:some.dotted.name#config.cfg would be clearer: the > "hierarchical" URL form is misleading. How so? The .cfg could be a file in a folder in the egg, same as a template for bfg... > We could even adopt the > convention that such config files needed to be registered as entry > points under some given section, e.g.: I don't see what purpose this would serve. I was ancticipating them being just like any other package_resource... Chris From tseaver at palladion.com Mon May 17 17:37:04 2010 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 17 May 2010 11:37:04 -0400 Subject: [Distutils] [buildout] extending a .cfg supplied by another package In-Reply-To: <4BF12156.5080508@simplistix.co.uk> References: <4BEA9C69.8060602@simplistix.co.uk> <87zl02c4bd.fsf@elzar.ws.whq.gocept.com> <4BF12156.5080508@simplistix.co.uk> Message-ID: <4BF162A0.2040504@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote: > Tres Seaver wrote: >>>> project://some.project/aconfig.cfg >>> I'd propose egg://some.dotted.name/config.cfg >> I think egg:some.dotted.name#config.cfg would be clearer: the >> "hierarchical" URL form is misleading. > > How so? The .cfg could be a file in a folder in the egg, same as a > template for bfg... The hierarchical form URLs (containing ';//') imply an "authority" section (aka "nethost"). >> We could even adopt the >> convention that such config files needed to be registered as entry >> points under some given section, e.g.: > > I don't see what purpose this would serve. I was ancticipating them > being just like any other package_resource... Unlike other hose config files are almost never normal "package data" -- they are practically always kept at the top level of the project checkout, along with other "packaging metdata" like setup.py, README.txt, etc. I just checked, and sure enough, *none* of those files are present in "installed" eggs, which makes the whole idea a good deal less practical / valuable: neither your path-based syntax nor my entry point is going to have the file available. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvxYqAACgkQ+gerLs4ltQ7A/QCfaLJ7BlXHEq+eYif15MB7kk0S tYAAoIJKiSKcxoSEyZvKGd1uX8DeSKmF =FDY5 -----END PGP SIGNATURE----- From chris at simplistix.co.uk Mon May 17 18:02:22 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 17 May 2010 17:02:22 +0100 Subject: [Distutils] [buildout] extending a .cfg supplied by another package In-Reply-To: <4BF162A0.2040504@palladion.com> References: <4BEA9C69.8060602@simplistix.co.uk> <87zl02c4bd.fsf@elzar.ws.whq.gocept.com> <4BF12156.5080508@simplistix.co.uk> <4BF162A0.2040504@palladion.com> Message-ID: <4BF1688E.209@simplistix.co.uk> Tres Seaver wrote: > Unlike other hose config files are almost never normal "package data" -- > they are practically always kept at the top level of the project > checkout, along with other "packaging metdata" like setup.py, > README.txt, etc. I just checked, and sure enough, *none* of those files > are present in "installed" eggs, which makes the whole idea a good deal > less practical / valuable: neither your path-based syntax nor my entry > point is going to have the file available. Indeed, this is an annoying bug I bumped into before: http://mail.python.org/pipermail/distutils-sig/2008-August/009900.html ...I remember it being an insane interaction of add and remvoe lists somewhere deep in distutils :-/ Still, if the .cfg was in a folder, then it should be possible to treat it as a normal package_resource, which would work fine with my pattern, where the root buildout.cfg is for that package only, and anything designed to be extended (generally a base.cfg and a versions.cfg) could be in a subfolder. When I get a chance (which may be never!) I'm going to have a go at doing my idea (enhanced by Jim's suggestion of urllib2 url handlers!) as a buildout extension. Thoughts welcome! cheers, Chris From tseaver at palladion.com Mon May 17 17:37:04 2010 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 17 May 2010 11:37:04 -0400 Subject: [Distutils] [buildout] extending a .cfg supplied by another package In-Reply-To: <4BF12156.5080508@simplistix.co.uk> References: <4BEA9C69.8060602@simplistix.co.uk> <87zl02c4bd.fsf@elzar.ws.whq.gocept.com> <4BF12156.5080508@simplistix.co.uk> Message-ID: <4BF162A0.2040504@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote: > Tres Seaver wrote: >>>> project://some.project/aconfig.cfg >>> I'd propose egg://some.dotted.name/config.cfg >> I think egg:some.dotted.name#config.cfg would be clearer: the >> "hierarchical" URL form is misleading. > > How so? The .cfg could be a file in a folder in the egg, same as a > template for bfg... The hierarchical form URLs (containing ';//') imply an "authority" section (aka "nethost"). >> We could even adopt the >> convention that such config files needed to be registered as entry >> points under some given section, e.g.: > > I don't see what purpose this would serve. I was ancticipating them > being just like any other package_resource... Unlike other hose config files are almost never normal "package data" -- they are practically always kept at the top level of the project checkout, along with other "packaging metdata" like setup.py, README.txt, etc. I just checked, and sure enough, *none* of those files are present in "installed" eggs, which makes the whole idea a good deal less practical / valuable: neither your path-based syntax nor my entry point is going to have the file available. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvxYqAACgkQ+gerLs4ltQ7A/QCfaLJ7BlXHEq+eYif15MB7kk0S tYAAoIJKiSKcxoSEyZvKGd1uX8DeSKmF =FDY5 -----END PGP SIGNATURE----- From sridharr at activestate.com Mon May 17 19:35:33 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Mon, 17 May 2010 10:35:33 -0700 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: References: Message-ID: <37936094-9EF4-4AFB-A81C-26661A83E1F3@activestate.com> On 2010-05-15, at 1:04 AM, Tarek Ziad? wrote: > IIRC you have a commit access so you can fix bugs too ;) I don't think I do: $ hg ci virtualenv.py -m "Update our local copy of virtualenv.py; now plays well with py2.7 except for --no-site-packages in bootstrap.sh (this is also a test commit to see if I really have commit access)" pushing to https://srid at bitbucket.org/tarek/distribute searching for changes http authorization required realm: Bitbucket.org HTTP user: srid password: abort: authorization failed warning: commit.autopush hook exited with status 255 .. but would be glad to start with committing minor fixes. > And I'd be glad to give you a pypi access and to show you how to do releases I am not yet prepared to do *releases* though. :-) -srid From ziade.tarek at gmail.com Mon May 17 23:56:30 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 17 May 2010 23:56:30 +0200 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: <37936094-9EF4-4AFB-A81C-26661A83E1F3@activestate.com> References: <37936094-9EF4-4AFB-A81C-26661A83E1F3@activestate.com> Message-ID: On Mon, May 17, 2010 at 7:35 PM, Sridhar Ratnakumar wrote: > > On 2010-05-15, at 1:04 AM, Tarek Ziad? wrote: > >> ?IIRC you have a commit access so you can fix bugs too ;) > > I don't think I do: Sorry I thought you were in. "srid" added ! [..] >> And I'd be glad to give you a pypi access and to show you how to do releases > > I am not yet prepared to do *releases* though. :-) Well, let me know if you want to... ;) -- Tarek Ziad? | http://ziade.org From itconsense at gmail.com Tue May 18 10:59:42 2010 From: itconsense at gmail.com (Tom Gross) Date: Tue, 18 May 2010 10:59:42 +0200 Subject: [Distutils] pypi.python.org as single point of failure Message-ID: Hello together, because of the breakdown of pypi.python.org I found that this Python index is hardcoded in the distribute_setup.py-script. I tried to workaround the breakage of the index-server by changing my hosts-file pointing to a pypi-mirror found on http://www.coactivate.org/projects/pypi-mirroring/project-home, but the directory structure is different there, so it didn't work. I ran into this problem first by using the bootstrap-command of zc.buildout. this is probably the wrong list to discuss this, but I think it should be possible to *fully* replace pypi by a mirror for all setuptools, distribute, zc.buildout-variants by configuration. Cheers, -Tom From reinout at vanrees.org Tue May 18 15:10:09 2010 From: reinout at vanrees.org (Reinout van Rees) Date: Tue, 18 May 2010 15:10:09 +0200 Subject: [Distutils] pypi.python.org as single point of failure In-Reply-To: References: Message-ID: On 05/18/2010 10:59 AM, Tom Gross wrote: > because of the breakdown of pypi.python.org I found that this Python > index is hardcoded in the distribute_setup.py-script. > I tried to workaround the breakage of the index-server by changing my > hosts-file pointing to a pypi-mirror found on > http://www.coactivate.org/projects/pypi-mirroring/project-home, but the > directory structure is different there, so it didn't work. The difference is the lack of /simple in the directory structure, btw. So http://pypi.python.org/simple/ would be the same as http://pypi.zopyx.com/ . So the latter is without the /simple. Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Programmer at http://www.nelen-schuurmans.nl "Military engineers build missiles. Civil engineers build targets" From itconsense at gmail.com Tue May 18 15:23:56 2010 From: itconsense at gmail.com (Tom Gross) Date: Tue, 18 May 2010 15:23:56 +0200 Subject: [Distutils] pypi.python.org as single point of failure In-Reply-To: References: Message-ID: On 05/18/2010 03:10 PM, Reinout van Rees wrote: > On 05/18/2010 10:59 AM, Tom Gross wrote: > >> because of the breakdown of pypi.python.org I found that this Python >> index is hardcoded in the distribute_setup.py-script. >> I tried to workaround the breakage of the index-server by changing my >> hosts-file pointing to a pypi-mirror found on >> http://www.coactivate.org/projects/pypi-mirroring/project-home, but the >> directory structure is different there, so it didn't work. > > The difference is the lack of /simple in the directory structure, btw. > So http://pypi.python.org/simple/ would be the same as > http://pypi.zopyx.com/ . So the latter is without the /simple. > I know, but that's not the issue. It is that http://pypi.python.org/packages/source/d/distribute/ is hardcoded in http://python-distribute.org/distribute_setup.py The location of the distribute-egg should be a configurable option of distribute_setup.py -Tom > > Reinout > From ziade.tarek at gmail.com Wed May 19 14:07:07 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 19 May 2010 14:07:07 +0200 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: References: Message-ID: On Sat, May 15, 2010 at 12:30 AM, Sridhar Ratnakumar wrote: > May I call your attention to this bug? > > ?http://bitbucket.org/tarek/distribute/issue/151/ > > 0.6.12 broke Windows installation. Are we not planning a new release? The problem was fixed. Can you check on your side that everything is running fine, so I can do a new release You can clone the repo and get in the 0.6-maintenance branch, Thanks, > -srid > > -- Tarek Ziad? | http://ziade.org From sridharr at activestate.com Wed May 19 23:44:18 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 19 May 2010 14:44:18 -0700 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: References: Message-ID: <4BF45BB2.9050208@activestate.com> On 5/19/2010 5:07 AM, Tarek Ziad? wrote: > On Sat, May 15, 2010 at 12:30 AM, Sridhar Ratnakumar > wrote: >> > May I call your attention to this bug? >> > >> > http://bitbucket.org/tarek/distribute/issue/151/ >> > >> > 0.6.12 broke Windows installation. Are we not planning a new release? > The problem was fixed. Can you check on your side that everything is > running fine, > so I can do a new release > > You can clone the repo and get in the 0.6-maintenance branch, 0.6-maintenance installs fine on Python 2.7: http://gist.github.com/406877 I have also verified further by using the installed easy_install to install a sample package. Looks good to me, at least on Python trunk (2.7) compiled a week ago. Similarly, 0.6-maintenance also installs successfully on Python 2.6.5. -srid From ziade.tarek at gmail.com Wed May 19 23:51:31 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 19 May 2010 23:51:31 +0200 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: <4BF45BB2.9050208@activestate.com> References: <4BF45BB2.9050208@activestate.com> Message-ID: On Wed, May 19, 2010 at 11:44 PM, Sridhar Ratnakumar wrote: > On 5/19/2010 5:07 AM, Tarek Ziad? wrote: >> >> On Sat, May 15, 2010 at 12:30 AM, Sridhar Ratnakumar >> ?wrote: >>> >>> > ?May I call your attention to this bug? >>> > >>> > ? ?http://bitbucket.org/tarek/distribute/issue/151/ >>> > >>> > ?0.6.12 broke Windows installation. Are we not planning a new release? >> >> The problem was fixed. Can you check on your side that everything is >> running fine, >> so I can do a new release >> >> You can clone the repo and get in the 0.6-maintenance branch, > > 0.6-maintenance installs fine on Python 2.7: > ?http://gist.github.com/406877 > > I have also verified further by using the installed easy_install to install > a sample package. > > Looks good to me, at least on Python trunk (2.7) compiled a week ago. > > Similarly, 0.6-maintenance also installs successfully on Python 2.6.5. > Ronny has fixed the .local / ENABLE_USER_SITE flag if you want to give it a shot too > -srid > -- Tarek Ziad? | http://ziade.org From sridharr at activestate.com Wed May 19 23:53:29 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 19 May 2010 14:53:29 -0700 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: References: <4BF45BB2.9050208@activestate.com> Message-ID: <277D35A4-B772-465C-9EDF-2E2B7176E906@activestate.com> On 2010-05-19, at 2:51 PM, Tarek Ziad? wrote: > On Wed, May 19, 2010 at 11:44 PM, Sridhar Ratnakumar > wrote: >> On 5/19/2010 5:07 AM, Tarek Ziad? wrote: >>> >>> On Sat, May 15, 2010 at 12:30 AM, Sridhar Ratnakumar >>> wrote: >>>> >>>>> May I call your attention to this bug? >>>>> >>>>> http://bitbucket.org/tarek/distribute/issue/151/ >>>>> >>>>> 0.6.12 broke Windows installation. Are we not planning a new release? >>> >>> The problem was fixed. Can you check on your side that everything is >>> running fine, >>> so I can do a new release >>> >>> You can clone the repo and get in the 0.6-maintenance branch, >> >> 0.6-maintenance installs fine on Python 2.7: >> http://gist.github.com/406877 >> >> I have also verified further by using the installed easy_install to install >> a sample package. >> >> Looks good to me, at least on Python trunk (2.7) compiled a week ago. >> >> Similarly, 0.6-maintenance also installs successfully on Python 2.6.5. >> > > Ronny has fixed the .local / ENABLE_USER_SITE flag if you want to give > it a shot too You mean "easy_install --user"? Yup, that too works as expected. -srid From sridharr at activestate.com Thu May 20 00:10:09 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 19 May 2010 15:10:09 -0700 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: <277D35A4-B772-465C-9EDF-2E2B7176E906@activestate.com> References: <4BF45BB2.9050208@activestate.com> <277D35A4-B772-465C-9EDF-2E2B7176E906@activestate.com> Message-ID: <48671EFF-3C05-4120-960E-0A5C46CE025B@activestate.com> On 2010-05-19, at 2:53 PM, Sridhar Ratnakumar wrote: >> Ronny has fixed the .local / ENABLE_USER_SITE flag if you want to give >> it a shot too > > You mean "easy_install --user"? Yup, that too works as expected. If you were rather referring to this bug: http://bitbucket.org/tarek/distribute/issue/150/ - no, it has not been fixed in 0.6-maint. -srid From Ronny.Pfannschmidt at gmx.de Thu May 20 00:47:36 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 20 May 2010 00:47:36 +0200 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: <48671EFF-3C05-4120-960E-0A5C46CE025B@activestate.com> References: <4BF45BB2.9050208@activestate.com> <277D35A4-B772-465C-9EDF-2E2B7176E906@activestate.com> <48671EFF-3C05-4120-960E-0A5C46CE025B@activestate.com> Message-ID: <1274309256.8238.5.camel@klappe2> On Wed, 2010-05-19 at 15:10 -0700, Sridhar Ratnakumar wrote: > On 2010-05-19, at 2:53 PM, Sridhar Ratnakumar wrote: > > >> Ronny has fixed the .local / ENABLE_USER_SITE flag if you want to give > >> it a shot too > > > > You mean "easy_install --user"? Yup, that too works as expected. > > If you were rather referring to this bug: http://bitbucket.org/tarek/distribute/issue/150/ - no, it has not been fixed in 0.6-maint. > It would be nice to get some more help for debugging that one, i lack a affected system/idea how to really reproduce all simple tests with a virtualenv point me to works fine now (as i added the code to respect ENABLE_USER_SITE) -- Ronny From sridharr at activestate.com Thu May 20 18:34:10 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 20 May 2010 09:34:10 -0700 Subject: [Distutils] 2.6 replacement for sysconfig.get_paths(scheme)[path]? Message-ID: <195A9C26-D15E-4074-9E3F-0B959608D0D9@activestate.com> Python 2.7 introduced the `sysconfig` module which has a function `get_paths` that returns the fully variable substituted path for the given scheme. This doesn't seem like possible on 2.6 using `distutils.sysconfig` - as variable substitution is done deep in a class method, and not exposed as a public API. At the moment I am writing code like this: scheme = 'nt' sys.platform == 'win32' else 'unix' scheme += ('_user' if usersite else ( '_prefix' if scheme == 'unix' else '')) template = INSTALL_SCHEMES[scheme][path] # hardcoded template substitution (distutils API does not provide # this) template = template.replace('$base', base_dir) template = template.replace('$py_version_short', pyver) if '$user' in template: template = template.replace('$usersite', site.USER_SITE) template = template.replace('$userbase', site.USER_BASE) assert '$' not in template, \ 'Unsubstituded variables left in "%s"' % template return template Can this be simplified? -srid From sridharr at activestate.com Thu May 20 23:09:44 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 20 May 2010 14:09:44 -0700 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: <1274309256.8238.5.camel@klappe2> References: <4BF45BB2.9050208@activestate.com> <277D35A4-B772-465C-9EDF-2E2B7176E906@activestate.com> <48671EFF-3C05-4120-960E-0A5C46CE025B@activestate.com> <1274309256.8238.5.camel@klappe2> Message-ID: <4BF5A518.7080603@activestate.com> On 5/19/2010 3:47 PM, Ronny Pfannschmidt wrote: > On Wed, 2010-05-19 at 15:10 -0700, Sridhar Ratnakumar wrote: >> > On 2010-05-19, at 2:53 PM, Sridhar Ratnakumar wrote: >> > >>>> > >> Ronny has fixed the .local / ENABLE_USER_SITE flag if you want to give >>>> > >> it a shot too >>> > > >>> > > You mean "easy_install --user"? Yup, that too works as expected. >> > >> > If you were rather referring to this bug:http://bitbucket.org/tarek/distribute/issue/150/ - no, it has not been fixed in 0.6-maint. >> > > It would be nice to get some more help for debugging that one, > i lack a affected system/idea how to really reproduce Install $package into your user site. I used "pypm install $package" for this, which by defaults puts the package in the user site. And then create a virtualenv, upgrade its setuptools to distribute-dev and run $virtualenv/Scripts/easy_install.exe $package. > all simple tests with a virtualenv point me to works fine now > (as i added the code to respect ENABLE_USER_SITE) I found out where the problem was. This patch fixes it for me: diff -r ab666b0eacbb setuptools/command/easy_install.py --- a/setuptools/command/easy_install.py Wed May 19 22:43:52 2010 +0200 +++ b/setuptools/command/easy_install.py Thu May 20 14:06:56 2010 -0700 @@ -1347,8 +1347,7 @@ site_lib = get_python_lib(plat_specific) if site_lib not in sitedirs: sitedirs.append(site_lib) - if sys.version >= "2.6": - import site + if HAS_USER_SITE and site.ENABLE_USER_SITE: sitedirs.append(site.USER_SITE) sitedirs = map(normalize_path, sitedirs) -srid From Ronny.Pfannschmidt at gmx.de Fri May 21 19:25:20 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 21 May 2010 19:25:20 +0200 Subject: [Distutils] Distribute 0.6.12 broken on Windows In-Reply-To: <4BF5A518.7080603@activestate.com> References: <4BF45BB2.9050208@activestate.com> <277D35A4-B772-465C-9EDF-2E2B7176E906@activestate.com> <48671EFF-3C05-4120-960E-0A5C46CE025B@activestate.com> <1274309256.8238.5.camel@klappe2> <4BF5A518.7080603@activestate.com> Message-ID: <1274462720.9589.3.camel@klappe2> On Thu, 2010-05-20 at 14:09 -0700, Sridhar Ratnakumar wrote: > On 5/19/2010 3:47 PM, Ronny Pfannschmidt wrote: > > On Wed, 2010-05-19 at 15:10 -0700, Sridhar Ratnakumar wrote: > >> > On 2010-05-19, at 2:53 PM, Sridhar Ratnakumar wrote: > >> > > >>>> > >> Ronny has fixed the .local / ENABLE_USER_SITE flag if you want to give > >>>> > >> it a shot too > >>> > > > >>> > > You mean "easy_install --user"? Yup, that too works as expected. > >> > > >> > If you were rather referring to this bug:http://bitbucket.org/tarek/distribute/issue/150/ - no, it has not been fixed in 0.6-maint. > >> > > > It would be nice to get some more help for debugging that one, > > i lack a affected system/idea how to really reproduce > > Install $package into your user site. I used "pypm install $package" for > this, which by defaults puts the package in the user site. And then > create a virtualenv, upgrade its setuptools to distribute-dev and run > $virtualenv/Scripts/easy_install.exe $package. > > > all simple tests with a virtualenv point me to works fine now > > (as i added the code to respect ENABLE_USER_SITE) > > I found out where the problem was. This patch fixes it for me: i think i found a more general fix of the issue, can you please try if http://bitbucket.org/RonnyPfannschmidt/distribute-0.6 works as expected -- Ronny > > diff -r ab666b0eacbb setuptools/command/easy_install.py > --- a/setuptools/command/easy_install.py Wed May 19 22:43:52 2010 > +0200 > +++ b/setuptools/command/easy_install.py Thu May 20 14:06:56 2010 > -0700 > @@ -1347,8 +1347,7 @@ > site_lib = get_python_lib(plat_specific) > if site_lib not in sitedirs: sitedirs.append(site_lib) > > - if sys.version >= "2.6": > - import site > + if HAS_USER_SITE and site.ENABLE_USER_SITE: > sitedirs.append(site.USER_SITE) > > sitedirs = map(normalize_path, sitedirs) > > -srid From manlio.perillo at gmail.com Sat May 22 17:55:16 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Sat, 22 May 2010 17:55:16 +0200 Subject: [Distutils] development egg and --find-links easy_install option Message-ID: <4BF7FE64.8040806@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. Reading easy_install documentation I recently discovered the - --find-links option, and I'm starting to experiment with it in order to create a tar archive with all the dependencies of a Python package. This is very useful when I have to deploy a package on a shared hosting, where can be a problem to compile extension modules. Unfortunately in my setup I use some packages (the main package being installed and one of its install requirements) in "develop" mode. These packages, moreover, are not available on PyPi. To summarize, I have a package A (to be installed in develop mode), that depends on package B, installed in develop mode. B, then, have additional install requirements, as "normal" packages. A have additional install requirements, as "normal" packages. When I run, from A source directory, the command: python setup.py develop -maxd deps/ an A.egg-link file is correctly copied to the `deps` directory. However, when processing B package, I get: Skipping development or system egg: B 0.1dev Reading http://pypi.python.org/simple/B/ ... Using the `-H none` option, will skip processing of B package and additional dependencies (including direct A dependencies). What is the reason why a development egg is being skipped? What I want is that in `deps` directory should copied all "normal" packages, as eggs, that are install requirements for package A or, indirectly, package B. Is this possible? Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv3/mQACgkQscQJ24LbaUSpbACfcp0gFJhxtlZpjR5D5XQg7q8J FKwAn1ddZhdzQorR+CxGSt30sW+qRKuW =pJ0V -----END PGP SIGNATURE----- From pje at telecommunity.com Sat May 22 19:25:33 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 22 May 2010 13:25:33 -0400 Subject: [Distutils] development egg and --find-links easy_install option In-Reply-To: <4BF7FE64.8040806@gmail.com> References: <4BF7FE64.8040806@gmail.com> Message-ID: <20100522172538.72B763A4061@sparrow.telecommunity.com> At 05:55 PM 5/22/2010 +0200, Manlio Perillo wrote: >What is the reason why a development egg is being skipped? Because it can't be copied (currently), and the -a flag in your -maxd means "--always-copy" -- that is, copy the egg, even if it's on sys.path. >What I want is that in `deps` directory should copied all "normal" >packages, as eggs, that are install requirements for package A or, >indirectly, package B. > >Is this possible? Yes, just build eggs for your local packages and place them somewhere accessible to --find-links. Alternatively, ensure that all needed packages are on sys.path, and use -mxd instead of -maxd. (i.e., allow setuptools to *not* copy things that are already available on sys.path.) From manlio.perillo at gmail.com Sat May 22 22:39:34 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Sat, 22 May 2010 22:39:34 +0200 Subject: [Distutils] development egg and --find-links easy_install option In-Reply-To: <20100522172538.72B763A4061@sparrow.telecommunity.com> References: <4BF7FE64.8040806@gmail.com> <20100522172538.72B763A4061@sparrow.telecommunity.com> Message-ID: <4BF84106.2040904@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 P.J. Eby ha scritto: > At 05:55 PM 5/22/2010 +0200, Manlio Perillo wrote: >> What is the reason why a development egg is being skipped? > > Because it can't be copied (currently), And I usually don't need it to be copied, since on the production server I will install these package by hand, from a Mercurial working copy. > and the -a flag in your -maxd > means "--always-copy" -- that is, copy the egg, even if it's on sys.path. > >> What I want is that in `deps` directory should copied all "normal" >> packages, as eggs, that are install requirements for package A or, >> indirectly, package B. >> >> Is this possible? > > Yes, just build eggs for your local packages and place them somewhere > accessible to --find-links. Alternatively, ensure that all needed > packages are on sys.path, and use -mxd instead of -maxd. (i.e., allow > setuptools to *not* copy things that are already available on sys.path.) > But the idea was to copy all required packages to a common directory, so that I can create a tar archive from this directory and use it on the production server. And to have this done automatically, without having to build each required package. The needed packages are usually already on sys.path, since they have been installed the first time I executed "python setup.py develop" for my package, on my development machine. The trivial solution is of course to not use "develop" command, and to build a normal egg. I would like that setuptools will just copy the .egg-info file, *making sure* to use a relative path instead of an absolute path. As an example, using virtualenv I have this layout: A/ env/ bin/ include/ lib/ hg/ B/ deps/ setup.py ... `env` is the virtual environment. Inside the `hg` directory I have clones of the packages required by A and that are still under heavy development; these packages are installed using "develop" command. Inside the `deps` directory, I have all the eggs required by A and B packages. I copy all required eggs in this directory, using `-maxd` options, on my development machine. Then I create a tar archive, and, on the production server, I recreate the directory. The virtual environment is recreated each time, I do not use the - --relocatable option. If `-maxd` will make the B.egg-info file relative, then all I need to do is to `python setup.py develop` inside A directory, and B package should be correctly available. Is this possible? Thanks and sorry for the long message Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv4QQYACgkQscQJ24LbaUSJqwCeMEuPGWJHyUyVsSFuevw1V/n1 FBUAnA0XcskGAKwTbmN5tqEVLhhMtG3L =G3Dn -----END PGP SIGNATURE----- From pje at telecommunity.com Sun May 23 00:01:25 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 22 May 2010 18:01:25 -0400 Subject: [Distutils] development egg and --find-links easy_install option In-Reply-To: <4BF84106.2040904@gmail.com> References: <4BF7FE64.8040806@gmail.com> <20100522172538.72B763A4061@sparrow.telecommunity.com> <4BF84106.2040904@gmail.com> Message-ID: <20100522220129.CC9943A4061@sparrow.telecommunity.com> At 10:39 PM 5/22/2010 +0200, Manlio Perillo wrote: >The trivial solution is of course to not use "develop" command, and to >build a normal egg. Right. The slightly-less-trivial version is to make sure your source is in subversion, and add svn: links to your --find-links. >If `-maxd` will make the B.egg-info file relative, then all I need to do >is to `python setup.py develop` inside A directory, and B package should >be correctly available. > > >Is this possible? The -a in -maxd means that you must have either a source distribution (e.g. an sdist .tgz, svn: link, etc.) or an .egg. It cannot do .egg-info at the moment (although when it grows PEP 376 support in 0.7 it probably will). The easiest way to do this would be to add something like: svn://myprivateserver/svndir/projectname/trunk#egg=projectname-dev To your --find-links, and set your requirements to "projectname>=NN.NN,==dev", where NN.NN is the version you actually need, and the ==dev part allows it to pull the version from SVN and install it. (Of course, if you don't have or use svn, this doesn't work; at the moment there isn't support for adding new URL protocols to easy_install, although it's on my list for 0.7.) If this way doesn't work, you'll need to either use sdist, or make sure you do an egg install of your local packages before attempting to build a release. Btw, there's still one MORE way to do this: easy_install -maxd targetdir sourcedir1 sourcedir2 ... Where sourcedir1, sourcedir2, etc. are local paths to project directories containing setup.py files. They'll all be built eggs and put in the targetdir. (If you do this, however, you should list the dependencies *first*, rather than last, otherwise it will not be able to resolve them, if you are still only using 'develop' installs there.) From manlio.perillo at gmail.com Sun May 23 00:41:40 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Sun, 23 May 2010 00:41:40 +0200 Subject: [Distutils] development egg and --find-links easy_install option In-Reply-To: <20100522220129.CC9943A4061@sparrow.telecommunity.com> References: <4BF7FE64.8040806@gmail.com> <20100522172538.72B763A4061@sparrow.telecommunity.com> <4BF84106.2040904@gmail.com> <20100522220129.CC9943A4061@sparrow.telecommunity.com> Message-ID: <4BF85DA4.3000900@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 P.J. Eby ha scritto: > At 10:39 PM 5/22/2010 +0200, Manlio Perillo wrote: >> The trivial solution is of course to not use "develop" command, and to >> build a normal egg. > > Right. The slightly-less-trivial version is to make sure your source is > in subversion, and add svn: links to your --find-links. > Unfortunately I no more use Subversion. Do you plan to add full support to other VCS, or should I switch to a setuptools fork? > >> If `-maxd` will make the B.egg-info file relative, then all I need to do >> is to `python setup.py develop` inside A directory, and B package should >> be correctly available. >> >> >> Is this possible? > > The -a in -maxd means that you must have either a source distribution > (e.g. an sdist .tgz, svn: link, etc.) or an .egg. It cannot do > .egg-info at the moment (although when it grows PEP 376 support in 0.7 > it probably will). > Is it .egg-info or .egg-link ? If it is an .egg-link, then this could be copied as with normal eggs, of course promising (to setuptools) that the path is linked to a valid directory. I tried right now, and it is possible to specify a relative path in the .egg-link file. > [...] > Btw, there's still one MORE way to do this: > > easy_install -maxd targetdir sourcedir1 sourcedir2 ... > > Where sourcedir1, sourcedir2, etc. are local paths to project > directories containing setup.py files. They'll all be built eggs and > put in the targetdir. > > (If you do this, however, you should list the dependencies *first*, > rather than last, otherwise it will not be able to resolve them, if you > are still only using 'develop' installs there.) > What do you mean by "first"? Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv4XaQACgkQscQJ24LbaUSkvgCgiAoOIl7oVAJhMOqsndAQ2wk1 Lj4AnjQz47/UT80gGYtudjSbUd6a82ZY =sB+U -----END PGP SIGNATURE----- From ygingras at ygingras.net Sun May 23 03:46:54 2010 From: ygingras at ygingras.net (Yannick Gingras) Date: Sat, 22 May 2010 21:46:54 -0400 Subject: [Distutils] Script for easy access to the coverage report on distutils2 Message-ID: <201005222146.55404.ygingras@ygingras.net> Greetings Packagers, as part of our sprinting effort at Montr?al-Python, we found it relevant to give an easy access to the test coverage reports to sprinters because it really helps them to see what needs more testing and when kind of breakage then can expect the tests to catch. I tried to use nose but the naming convention used seem to trigger way too many false positives so I'm afraid that we have to rule distutils2 as nose-incompatible. I could add a series of commands to our wiki, which is good for us but not very helpful to other distutils2 hackers, or I could integrate coverage in the current `runtests.py` test runner. Is there anyone one against adding a --with coverage option to `runtests.py`? Cheers, -- Yannick Gingras http://ygingras.net http://montrealpython.org -- lead organizer -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From ygingras at ygingras.net Sun May 23 06:02:46 2010 From: ygingras at ygingras.net (Yannick Gingras) Date: Sun, 23 May 2010 00:02:46 -0400 Subject: [Distutils] Script for easy access to the coverage report on distutils2 In-Reply-To: <201005222146.55404.ygingras@ygingras.net> References: <201005222146.55404.ygingras@ygingras.net> Message-ID: <201005230002.47357.ygingras@ygingras.net> > Is there anyone one against adding a --with coverage option to > `runtests.py`? Please see the proof of concept patch in attachment. -- Yannick Gingras http://ygingras.net http://montrealpython.org -- lead organizer -------------- next part -------------- A non-text attachment was scrubbed... Name: run-test-with-coverage.diff Type: text/x-patch Size: 2898 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From pje at telecommunity.com Sun May 23 06:32:32 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 23 May 2010 00:32:32 -0400 Subject: [Distutils] development egg and --find-links easy_install option In-Reply-To: <4BF85DA4.3000900@gmail.com> References: <4BF7FE64.8040806@gmail.com> <20100522172538.72B763A4061@sparrow.telecommunity.com> <4BF84106.2040904@gmail.com> <20100522220129.CC9943A4061@sparrow.telecommunity.com> <4BF85DA4.3000900@gmail.com> Message-ID: <20100523043234.3E8A23A4061@sparrow.telecommunity.com> At 12:41 AM 5/23/2010 +0200, Manlio Perillo wrote: >P.J. Eby ha scritto: > > At 10:39 PM 5/22/2010 +0200, Manlio Perillo wrote: > >> The trivial solution is of course to not use "develop" command, and to > >> build a normal egg. > > > > Right. The slightly-less-trivial version is to make sure your source is > > in subversion, and add svn: links to your --find-links. > > > >Unfortunately I no more use Subversion. >Do you plan to add full support to other VCS, or should I switch to a >setuptools fork? As I mentioned, support for URL prefix plugins will be added in 0.7. (Not necessarily in the first alpha though.) > >> If `-maxd` will make the B.egg-info file relative, then all I need to do > >> is to `python setup.py develop` inside A directory, and B package should > >> be correctly available. > >> > >> > >> Is this possible? > > > > The -a in -maxd means that you must have either a source distribution > > (e.g. an sdist .tgz, svn: link, etc.) or an .egg. It cannot do > > .egg-info at the moment (although when it grows PEP 376 support in 0.7 > > it probably will). > > > >Is it .egg-info or .egg-link ? First off, there really isn't any sane meaning of 'develop' in the context of a -maxd tarball. A -maxd tarball needs .egg files or directories, or else you should just use pip to build a bundle or whatever it's called in pip. >What do you mean by "first"? I mean, "easy_install -maxd targetdir dependencysource dependersource" - the dependency source directories are to be listed before the source directories that depend on them. All things considered, this is probably the easiest way for you to build a tarball-deployable directory from local source packages that are not indexed on PyPI or pre-built. (I don't know whether pip can handle this use case; i.e., I don't know if it will take source directories as arguments.) From manlio.perillo at gmail.com Sun May 23 16:24:49 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Sun, 23 May 2010 16:24:49 +0200 Subject: [Distutils] development egg and --find-links easy_install option In-Reply-To: <20100523043234.3E8A23A4061@sparrow.telecommunity.com> References: <4BF7FE64.8040806@gmail.com> <20100522172538.72B763A4061@sparrow.telecommunity.com> <4BF84106.2040904@gmail.com> <20100522220129.CC9943A4061@sparrow.telecommunity.com> <4BF85DA4.3000900@gmail.com> <20100523043234.3E8A23A4061@sparrow.telecommunity.com> Message-ID: <4BF93AB1.3010803@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 P.J. Eby ha scritto: > [...] >> > >> > The -a in -maxd means that you must have either a source distribution >> > (e.g. an sdist .tgz, svn: link, etc.) or an .egg. It cannot do >> > .egg-info at the moment (although when it grows PEP 376 support in 0.7 >> > it probably will). >> > >> >> Is it .egg-info or .egg-link ? > > First off, there really isn't any sane meaning of 'develop' in the > context of a -maxd tarball. A -maxd tarball needs .egg files or > directories, or else you should just use pip to build a bundle or > whatever it's called in pip. > pip bundles only contain source distributions. So, it only solve the "I don't have network access" problem, but not the "I can not compile extension modules" problem. Looking at setuptools source code, I think I have solved the problem. The easy_install command when calling the `package_index.fetch_distribution` function, will set the `develop_ok` parameter to False, when `always_copy` option is True. I patched setuptools to always set the parameter to True, and now doing: python setup.py develop -H none -amxd deps do what I want. System and development eggs will no more be skipped, thus causing a DistutilsError exception to be raised; instead the currently installed version will be used. Since I always use `virtualenv --no-site-packages`, all non standard modules will be copied. I will probably implement a custom "bundle" distutils command, that will do this, creating an "eggs bundle" archive (using the sdist --formats option). Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv5OrEACgkQscQJ24LbaUThiwCfVp3qJY1xu5JedxHVzFoGScek lAAAn3zdVwYiyzp8v7xTg57GjR3DX7aB =ctTd -----END PGP SIGNATURE----- From ziade.tarek at gmail.com Sun May 23 18:41:36 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 23 May 2010 18:41:36 +0200 Subject: [Distutils] Script for easy access to the coverage report on distutils2 In-Reply-To: <201005222146.55404.ygingras@ygingras.net> References: <201005222146.55404.ygingras@ygingras.net> Message-ID: On Sun, May 23, 2010 at 3:46 AM, Yannick Gingras wrote: > > Greetings Packagers, > ?as part of our sprinting effort at Montr?al-Python, we found it > relevant to give an easy access to the test coverage reports to > sprinters because it really helps them to see what needs more testing > and when kind of breakage then can expect the tests to catch. > > I tried to use nose but the naming convention used seem to trigger way > too many false positives so I'm afraid that we have to rule distutils2 > as nose-incompatible. > > I could add a series of commands to our wiki, which is good for us but > not very helpful to other distutils2 hackers, or I could integrate > coverage in the current `runtests.py` test runner. ?Is there anyone > one against adding a --with coverage option to `runtests.py`? Good idea, let me know when it's up in your clone, and I'll review/pull it Notice that this script is the basis for the soon-to-be-set Hudson CI server, so we will be able to run a daily coverage report of some sort we can publish -- if you take care of making it possible to output HTML. Thanks, Tarek > > Cheers, > > -- > Yannick Gingras > http://ygingras.net > http://montrealpython.org -- lead organizer > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -- Tarek Ziad? | http://ziade.org From barry at python.org Mon May 24 22:05:11 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 24 May 2010 16:05:11 -0400 Subject: [Distutils] Script for easy access to the coverage report on distutils2 In-Reply-To: References: <201005222146.55404.ygingras@ygingras.net> Message-ID: <20100524160511.4a8c208d@heresy> On May 23, 2010, at 06:41 PM, Tarek Ziad? wrote: >Notice that this script is the basis for the soon-to-be-set Hudson CI server, >so we will be able to run a daily coverage report of some sort we can >publish -- if you >take care of making it possible to output HTML. Tarek, Can you describe this a bit more? Did I miss some discussion on distutils-sig about it? What I really want is a CI system that could run over all the packages on the Cheeseshop (or any other collection of Python packages, e.g. say in a Debian/Ubuntu repository), and run a standard test suite, providing some feedback as to the general health of the package on various versions of Python. I think a standard interface should be 'python setup.py test'. Packages don't have adhere to this standard but if they did, we could perhaps offer some nice carrots, like a star rating on Cheeseshop, detailed reports, etc. If something like this doesn't exist, I'm willing and available to work on it. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ziade.tarek at gmail.com Mon May 24 22:26:53 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 24 May 2010 22:26:53 +0200 Subject: [Distutils] Script for easy access to the coverage report on distutils2 In-Reply-To: <20100524160511.4a8c208d@heresy> References: <201005222146.55404.ygingras@ygingras.net> <20100524160511.4a8c208d@heresy> Message-ID: On Mon, May 24, 2010 at 10:05 PM, Barry Warsaw wrote: > On May 23, 2010, at 06:41 PM, Tarek Ziad? wrote: > >>Notice that this script is the basis for the soon-to-be-set Hudson CI server, >>so we will be able to run a daily coverage report of some sort we can >>publish -- if you >>take care of making it possible to output HTML. > > Tarek, > > Can you describe this a bit more? ?Did I miss some discussion on distutils-sig > about it? > > What I really want is a CI system that could run over all the packages on the > Cheeseshop (or any other collection of Python packages, e.g. say in a > Debian/Ubuntu repository), and run a standard test suite, providing some > feedback as to the general health of the package on various versions of > Python. ?I think a standard interface should be 'python setup.py test'. > Packages don't have adhere to this standard but if they did, we could perhaps > offer some nice carrots, like a star rating on Cheeseshop, detailed reports, > etc. > > If something like this doesn't exist, I'm willing and available to work on it. Hello Barry, Several things here : 1- a test command is going to be added in distutils2, and discussions have started on this feature, which is part of one GSOC project for Distutils2. See a summary of tasks here: http://wiki.python.org/moin/SummerOfCode/Distutils2 , and the GSOC who's who here: http://bitbucket.org/tarek/distutils2/wiki/GSoC_2010_teams. Michael, Titus and other ppl have started to discuss this test command, looking at setuptools' one and thinking about having "python setup.py test" as a standard. 2- there's a GSOC project I have proposed, and that is being mentored by Jesse Noller, that will work on creating a CI system that will run various tests on packages uploaded at PyPI. The hardest part here is to run in a protected environment that is "cleaned up" everytime a new package is tested. So we'll script Virtual Machines, and get reports back, and we will probably use the PubHubSomething interface for this. The goal of this project is to come up with a CI system over PyPI, and eventually ask the PSF to host it. Any help on 1/ or 2/ is welcome ! Now for distutils2 itself, what we said at the GSOC meeting is that we wanted to setup up a CI for its code, before the GSOC starts. It has a runtests.py script right now that looks like test_distutils from the stdlib, and use the test command interface as soon as it is added in distutils2. Yannick wants to have coverage, so he added an option to that script. Maybe, somehow, coverage and other metrics could be part of the test standard calling interface we want. I am not sure how this could look though, if we want to stay generic and avoid a dependency on a specific test tool (e.g. unittest vs nose vs ..) Regards Tarek > > -Barry > > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -- Tarek Ziad? | http://ziade.org From Ronny.Pfannschmidt at gmx.de Tue May 25 17:58:18 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 25 May 2010 17:58:18 +0200 Subject: [Distutils] on storing vcs version infos in sdists Message-ID: <1274803098.4199.7.camel@klappe2> hi, whats a good place to store version information from a vcs in a sdist, without needing to drop them into python files. i wrote a simple module that gets me version meta-data from hg repos/archives, using the same algorithms hg is using for its own version number generation. The nit in that is that i have to store it in a python file. I'd much rather put it somewhere else in case of sdists, but i have no idea where/how i should put it to fit with the setuptools/distribute standards. -- Ronny From pje at telecommunity.com Tue May 25 19:09:34 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 25 May 2010 13:09:34 -0400 Subject: [Distutils] on storing vcs version infos in sdists In-Reply-To: <1274803098.4199.7.camel@klappe2> References: <1274803098.4199.7.camel@klappe2> Message-ID: <20100525170927.5457F3A409D@sparrow.telecommunity.com> At 05:58 PM 5/25/2010 +0200, Ronny Pfannschmidt wrote: >hi, > >whats a good place to store version information from a vcs in a sdist, >without needing to drop them into python files. > >i wrote a simple module that gets me version meta-data from hg >repos/archives, >using the same algorithms hg is using for its own version number >generation. > >The nit in that is that i have to store it in a python file. >I'd much rather put it somewhere else in case of sdists, >but i have no idea where/how i should put it to fit with the >setuptools/distribute standards. Setuptools stores this information in the setup.cfg of an sdist, under: [egg_info] tag_build = .dev-r##### More precisely, what it does is take whatever was in tag_build before (either from setup.cfg or the command line) and then tack the revision info on the end, and stick it into the sdist's setup.cfg, with tag_svn_revision turned back off. So in the above example, if tag_build was originally .dev, then the -r#### bit got added on. If tag_build was empty, then it would've been just '-r####'. Anyway, if you store it in setup.cfg, then it will automatically get tacked on the end of the version specified in setup.py, and it will get used by all the version/dependency APIs, such as querying pkg_resources for a particular version of a project, or requesting a project's version. From barry at python.org Tue May 25 21:07:10 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 25 May 2010 15:07:10 -0400 Subject: [Distutils] Script for easy access to the coverage report on distutils2 In-Reply-To: References: <201005222146.55404.ygingras@ygingras.net> <20100524160511.4a8c208d@heresy> Message-ID: <20100525150710.103b5cdd@heresy> On May 24, 2010, at 10:26 PM, Tarek Ziad? wrote: >1- a test command is going to be added in distutils2, and discussions >have started on this feature, which is part of one GSOC project for >Distutils2. See a summary of tasks here: >http://wiki.python.org/moin/SummerOfCode/Distutils2 , and the GSOC >who's who here: >http://bitbucket.org/tarek/distutils2/wiki/GSoC_2010_teams. Michael, >Titus and other ppl have started to discuss this test command, looking >at setuptools' one and thinking about having "python setup.py test" as >a standard. Cool. I really like 'python setup.py test' as *a* standard interface for running a package's tests. This is available today in distribute, which for Ubuntu is what you actually get when you install setuptools. So today on Ubuntu I can run 'python setup.py test' if my setup.py has a 'test_suite' entry. You know all this of course. :) One of the other things I hope to work on soon is a Quickly[1] template for an opinionated creation of Python libraries and applications. By that I mean, we decide what best practices are for Python libraries, and we can define a template for the initial layout of such packages. I've talked about this to some people (though I really need to write up my ideas), but what I want is for a Python developer to be able to very easy and quickly run one command that gives them a layout which: * is a namespace package * 'python setup.py test' works (though initially does nothing) * has a base setup.py, README, license files * 'python setup.py docs' works (read: build_sphinx) * 'python setup.py publish' works (read: upload to pypi, upload docs, etc) * already has hooks for pylint, pyflakes, coverage * pre-loaded with 2to3 hooks * easy to add packaging for .deb, .rpm, etc. (extensible) * etc Being opinionated helps, with the goal of being appropriate for let's say 80-90% of Python packages out there. Complex stuff will always be complicated. Make the easy stuff stupid simple. So several ideas are involved: * One command from idea to fully fleshed out package * One command publishing * Common interface for automated testing 'health' of package * linting * test suite * easy to hook into CI * Cheeseshop can publish health statistics * Lots of carrots for package authors to DTRT >2- there's a GSOC project I have proposed, and that is being mentored >by Jesse Noller, that will work on creating a CI system that will run >various tests on packages uploaded at PyPI. The hardest part here is >to run in a protected environment that is "cleaned up" everytime a new >package is tested. So we'll script Virtual Machines, and get reports >back, and we will probably use the PubHubSomething interface for this. >The goal of this project is to come up with a CI system over PyPI, and >eventually ask the PSF to host it. This sounds excellent. We might be able to use some of the same infrastructure that e.g. Launchpad's build system uses for this. Running tests dovetails with the goal above of defining a standard way of testing the health of a package (test suite, plus lint/flakes, plus coverage). My current thinking for now is that 'python setup.py ' is the standard interface, where can be 'test', 'publish', 'lint', etc. I think this can be eventually improved with the appropriate 'python -m' commands. >Any help on 1/ or 2/ is welcome ! Yes, I am very interested in helping with both! >Now for distutils2 itself, what we said at the GSOC meeting is that we >wanted to setup up a CI for its code, before the GSOC starts. It has a >runtests.py script right now that looks like test_distutils from the >stdlib, and use the test command interface as soon as it is added in >distutils2. > >Yannick wants to have coverage, so he added an option to that script. >Maybe, somehow, coverage and other metrics could be part of the test >standard calling interface we want. I am not sure how this could look >though, if we want to stay generic and avoid a dependency on a >specific test tool (e.g. unittest vs nose vs ..) Again, for the thing I'm describing above, I am allowing myself to be opinionated. What I care about most is the command line interface, and well defined standards. I'd like it to be flexible enough so that experts can extend and customize, but most people and packages would be happy with the defaults. Anyway, I look forward to seeing the fruits of the GSoC work and participating where I can! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From Ronny.Pfannschmidt at gmx.de Wed May 26 00:19:47 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Wed, 26 May 2010 00:19:47 +0200 Subject: [Distutils] on storing vcs version infos in sdists In-Reply-To: <20100525170927.5457F3A409D@sparrow.telecommunity.com> References: <1274803098.4199.7.camel@klappe2> <20100525170927.5457F3A409D@sparrow.telecommunity.com> Message-ID: <1274825987.4199.20.camel@klappe2> On Tue, 2010-05-25 at 13:09 -0400, P.J. Eby wrote: > At 05:58 PM 5/25/2010 +0200, Ronny Pfannschmidt wrote: > >hi, > > > >whats a good place to store version information from a vcs in a sdist, > >without needing to drop them into python files. > > > >i wrote a simple module that gets me version meta-data from hg > >repos/archives, > >using the same algorithms hg is using for its own version number > >generation. > > > >The nit in that is that i have to store it in a python file. > >I'd much rather put it somewhere else in case of sdists, > >but i have no idea where/how i should put it to fit with the > >setuptools/distribute standards. > > Setuptools stores this information in the setup.cfg of an sdist, under: > > [egg_info] > tag_build = .dev-r##### > > More precisely, what it does is take whatever was in tag_build before > (either from setup.cfg or the command line) and then tack the > revision info on the end, and stick it into the sdist's setup.cfg, > with tag_svn_revision turned back off. > > So in the above example, if tag_build was originally .dev, then the > -r#### bit got added on. If tag_build was empty, then it would've > been just '-r####'. > > Anyway, if you store it in setup.cfg, then it will automatically get > tacked on the end of the version specified in setup.py, and it will > get used by all the version/dependency APIs, such as querying > pkg_resources for a particular version of a project, or requesting a > project's version. I don't just use the dev/build tag. I use a completely vcs based approach to releasing, which means the complete version is inferred from the vcs (not just the tag) Being at a revision that's tagged with a version number means i can skip the dev marker, anything else means i infer it via the distance in commits to the last version number tag + add the commit node id. Would storing the complete version in the build tag and passing setup.py a empty version string be a reasonable enough approach? On a side-note, i don't use svn for any of my projects, its hg for most of them. -- Ronny From cournape at gmail.com Wed May 26 00:44:15 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 26 May 2010 07:44:15 +0900 Subject: [Distutils] on storing vcs version infos in sdists In-Reply-To: <1274803098.4199.7.camel@klappe2> References: <1274803098.4199.7.camel@klappe2> Message-ID: On Wed, May 26, 2010 at 12:58 AM, Ronny Pfannschmidt wrote: > hi, > > whats a good place to store version information from a vcs in a sdist, > without needing to drop them into python files. What's the reason for not putting it in python files ? If the reason is to avoid hardcoding things, you can generate the python file. What I generally do is to "build" the version in setup.py, and generate a trivial __version.py module which is imported by the module to know its version. The __version.py is always generated everytime setup.py (so will be taken into account when doing sdist), and I protect the import of __version to have a default value, so that the package may be imported before any call to setup.py David From destrero at imavis.com Thu May 27 13:08:08 2010 From: destrero at imavis.com (Augusto Destrero) Date: Thu, 27 May 2010 13:08:08 +0200 Subject: [Distutils] "Out of source" bdist_egg, missing metadata Message-ID: <201005271308.08200.destrero@imavis.com> Hi, sorry if this email arrives twice. Now I'm registered to this list, so I can check that the email is delivered. :) I've a question on setuptools egg creation. I've a directory structure like this: python_libs/ |---- setup_foo | |---- setup.py |---- src | |---- foo | | |---- __init__.py | | |---- foo_module.py | |---- bar | | |---- __init__.py | | |---- bar_module.py | |---- common | | |---- __init__.py | | |---- common_module.py Basically I have a src directory containing three packages (foo, bar and common), and I have a setup_foo directory OUTSIDE src, where I want to build an egg containing foo and common packages, and NOT bar package. In the setup_foo/setup.py script I have something like this: # -*- coding: utf-8 -*- from setuptools import setup, find_packages src_dir = "../src" setup(name='foo', version='0.1', description="foo and common packages", packages=find_packages(src_dir, exclude=['bar', 'bar.*']), package_dir={'': src_dir}, zip_safe=False, install_requires=[ 'PasteDeploy', ], entry_points=""" [paste.app_factory] foo_app = foo.foo_module:app_factory common_app = common.common_module:app_factory """, ) You can see that I have two entry points (in this case for use with PasteDeploy in a wsgi project). If I build an egg with that file: $ cd setup_foo/ $ python setup.py bdist_egg I can find a foo.egg-info directory under src which contains all metadata: $ ls src/foo.egg-info/ dependency_links.txt entry_points.txt not-zip-safe PKG-INFO requires.txt SOURCES.txt top_level.txt But in the egg created under setup_foo/dist I don't have those metadata: $ cd setup_foo/dist/ $ unzip foo-0.1-py2.6.egg $ ls EGG-INFO/ not-zip-safe SOURCES.txt And of course, after installing that egg with easy_install, PasteDeploy can't find the entry points. Is this a setuptools bug or I'm doing something wrong? I use setuptools 0.6.10 distributed in Ubuntu Lucid. Thank you in advance for your help, and excuse me for the lenghty email! -- Augusto Destrero From cournape at gmail.com Thu May 27 15:53:57 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 27 May 2010 22:53:57 +0900 Subject: [Distutils] "Out of source" bdist_egg, missing metadata In-Reply-To: <201005271308.08200.destrero@imavis.com> References: <201005271308.08200.destrero@imavis.com> Message-ID: On Thu, May 27, 2010 at 8:08 PM, Augusto Destrero wrote: > Hi, sorry if this email arrives twice. Now I'm registered to this list, so I > can check that the email is delivered. :) > > I've a question on setuptools egg creation. I've a directory structure like > this: > > python_libs/ > |---- setup_foo > | ? ? |---- setup.py > |---- src > | ? ? |---- foo > | ? ? | ? ? |---- __init__.py > | ? ? | ? ? |---- foo_module.py > | ? ? |---- bar > | ? ? | ? ? |---- __init__.py > | ? ? | ? ? |---- bar_module.py > | ? ? |---- common > | ? ? | ? ? |---- __init__.py > | ? ? | ? ? |---- common_module.py I guess you have a reason to do so, but I would strongly advice against doing what you are doing. Distutils has no notion of src directory, and you will hit all kind of corner cases. > Basically I have a src directory containing three packages (foo, bar and > common), and I have a setup_foo directory OUTSIDE src, where I want to build > an egg containing foo and common packages, and NOT bar package. Split the packages, then, that's by far the best solution. Handle the aggregation at a higher level using one of the existing tool (either python tool, or system packaging tool depending on your preference). David From destrero at imavis.com Thu May 27 16:38:40 2010 From: destrero at imavis.com (Augusto Destrero) Date: Thu, 27 May 2010 16:38:40 +0200 Subject: [Distutils] "Out of source" bdist_egg, missing metadata In-Reply-To: References: <201005271308.08200.destrero@imavis.com> Message-ID: <201005271638.40829.destrero@imavis.com> Thank you for your suggestion. I solved changing my structure a little bit: python_libs/ |---- src | |---- setup_foo.py | |---- foo | | |---- __init__.py | | |---- foo_module.py | |---- bar | | |---- __init__.py | | |---- bar_module.py | |---- common | | |---- __init__.py | | |---- common_module.py And changed setup_foo.py like this: setup(name='foo', version='0.1', description="foo and common packages", packages=find_packages(exclude=['bar', 'bar.*']), zip_safe=False, ... Now everything works as expected. Maybe as you said an "out of source" build is not well supported. > Split the packages, then, that's by far the best solution. Handle the > aggregation at a higher level using one of the existing tool (either > python tool, or system packaging tool depending on your preference). > That's of course a good idea, but for now my project is made up of just three packages and I find this solution quite handy. Thanks again! -- Augusto Destrero From sridharr at activestate.com Thu May 27 19:25:33 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 27 May 2010 10:25:33 -0700 Subject: [Distutils] pkginfo python 3 port In-Reply-To: <4BFEA679.2010103@agendaless.com> References: <3702344C-48D9-4F9F-8C21-7CECEFA21466@activestate.com> <4BFD8CF0.1090201@agendaless.com> <30684B58-A48E-47B2-BBFC-C9FF0B2EB2BA@activestate.com> <4BFEA679.2010103@agendaless.com> Message-ID: <03B67BBD-832E-4253-93CD-D1AA0DA3A442@activestate.com> On 2010-05-27, at 10:06 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Sridhar Ratnakumar wrote: >> For a start, how about this patch? >> http://gist.github.com/415137 >> >> `email.parser` is available till 2.5; not sure about <=2.4 though. > > Your patch does break with Python 2.4, which I would like to continue > supporting. Maybe we can add some more conditional imports and glue > functions? Something like the attached patch. Sounds good. > BTW, your patch also breaks a unit test, due to differences in behavior > between the rfc822 parser and the email one: > > ====================================================================== > FAIL: test_parse_Description > (pkginfo.tests.test_distribution.DistributionTests) > - ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/tseaver/projects/parcel/src/pkginfo/pkginfo/tests/test_distribution.py", > line 104, in test_parse_Description > 'This package enables integration with\n' > AssertionError: 'This package enables integration with\n foo > servers.' != 'This package enables integration with\n foo servers.' Hmm. RFC 822 (rfc822) and RFC 2822 have different "unfolding" rules for multiple line header. Curiously http://docs.python.org/library/rfc822.html is advertised as "Parse RFC 2822 mail headers," and yet it seems to unfold multiple lines in accordance with RFC 822. I don't know what the solution to this problem is. I see that lines in the `Description` field (in PKG-INFO) are preceded with 8 spaces. So PKG-INFO is generated in accordance with RFC 822. Is there a way to parse a RFC 822 message in Python 3? -srid -------------- next part -------------- An HTML attachment was scrubbed... URL: From sridharr at activestate.com Thu May 27 20:48:41 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 27 May 2010 11:48:41 -0700 Subject: [Distutils] virtualenv python3 port Message-ID: <6FAD26D1-B06A-4CAC-981A-7E828D05E0B2@activestate.com> I would like to help with virtualenv Python 3 port. http://bitbucket.org/brandon/virtualenv3/ seems to be outdated (not in sync with core) and thus buggy. Does not even actually use Distribute as claimed. Brandon, would you be interested in merging and making a release if I provide a patch? Should I work on a separate branch or directly on yours? I intend to sync with the core, retaining 2to3 changes. -srid From brandon at rhodesmill.org Thu May 27 21:55:04 2010 From: brandon at rhodesmill.org (Brandon Craig Rhodes) Date: Thu, 27 May 2010 15:55:04 -0400 Subject: [Distutils] virtualenv python3 port In-Reply-To: <6FAD26D1-B06A-4CAC-981A-7E828D05E0B2@activestate.com> (Sridhar Ratnakumar's message of "Thu, 27 May 2010 11:48:41 -0700") References: <6FAD26D1-B06A-4CAC-981A-7E828D05E0B2@activestate.com> Message-ID: <8739xdm86f.fsf@rhodesmill.org> Sridhar Ratnakumar writes: > I would like to help with virtualenv Python 3 port. > http://bitbucket.org/brandon/virtualenv3/ seems to be outdated (not in > sync with core) and thus buggy. Does not even actually use Distribute > as claimed. How odd; it uses Distribute when I run it. What does it use when you run it? > Brandon, would you be interested in merging and making a release if I > provide a patch? Should I work on a separate branch or directly on > yours? I intend to sync with the core, retaining 2to3 changes. Yes! I'm always happy to accept new patches. None of my current projects are Python 3 ones at the moment, so I haven't been able to allocate any time to keeping virtualenv3 up to date. I look forward to seeing your patch! -- Brandon Craig Rhodes brandon at rhodesmill.org http://rhodesmill.org/brandon From cool-rr at cool-rr.com Thu May 27 22:40:48 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Thu, 27 May 2010 22:40:48 +0200 Subject: [Distutils] Having a Python package install itself under a different name Message-ID: Hello all! I want to ask you a question regarding something I want to do with my app. The reason I'm asking here is because I understand that Distribute somehow installs itself as "setuptools". (I've never looked into the details of it.) And I want something similar, so maybe you can advise me how to do it. I'm developing a package called `garlicsim`. The package is intended for Python 2.X, but I am also offerring Python 3 support on a different fork called `garlicsim_py3`.(1) So both of these packages live side by side on PyPI, and Python 3 users install `garlicsim_py3`, and Python 2 users install `garlicsim`. The problem is: When third party modules want to use garlicsim, they should have one package name to refer to, not two. Sure, they can do something like this: try: import garlicsim except ImportError: import garlicsim_py3 as garlicsim But I would prefer not to make the developers of these modules do this. Is there a way that `garlicsim_py3` will install itself under the alias `garlicsim`? What I want is for a Python 3 user to be able to `import garlicsim` and refer to the module all the time as `garlicsim`, but that it will really be `garlicsim_py3`. ---------- (1) I've reached the decision to support Python 3 on a fork instead of in the same code base; It's important for me that the code base will be clean, and I would really not want to introduce compatibilty hacks. Thanks, Ram Rachum. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu May 27 23:03:36 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 27 May 2010 17:03:36 -0400 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: Message-ID: <20100527210334.DCAE73A409D@sparrow.telecommunity.com> At 10:40 PM 5/27/2010 +0200, cool-RR wrote: >Is there a way that `garlicsim_py3` will install itself under the >alias `garlicsim`? What I want is for a Python 3 user to be able to >`import garlicsim` and refer to the module all the time as >`garlicsim`, but that it will really be `garlicsim_py3`. Why not just develop them both with the same package name? Your project name (in setup(name='...')) can still be different. From barry at python.org Thu May 27 23:10:44 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 27 May 2010 17:10:44 -0400 Subject: [Distutils] pkginfo python 3 port In-Reply-To: <03B67BBD-832E-4253-93CD-D1AA0DA3A442@activestate.com> References: <3702344C-48D9-4F9F-8C21-7CECEFA21466@activestate.com> <4BFD8CF0.1090201@agendaless.com> <30684B58-A48E-47B2-BBFC-C9FF0B2EB2BA@activestate.com> <4BFEA679.2010103@agendaless.com> <03B67BBD-832E-4253-93CD-D1AA0DA3A442@activestate.com> Message-ID: <20100527171044.7c218718@heresy> On May 27, 2010, at 10:25 AM, Sridhar Ratnakumar wrote: >Is there a way to parse a RFC 822 message in Python 3? If it's ASCII, you should have no problems using email.parser.Parser. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From cool-rr at cool-rr.com Thu May 27 23:16:44 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Thu, 27 May 2010 23:16:44 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: <20100527210334.DCAE73A409D@sparrow.telecommunity.com> References: <20100527210334.DCAE73A409D@sparrow.telecommunity.com> Message-ID: On Thu, May 27, 2010 at 11:03 PM, P.J. Eby wrote: > At 10:40 PM 5/27/2010 +0200, cool-RR wrote: > >> Is there a way that `garlicsim_py3` will install itself under the alias >> `garlicsim`? What I want is for a Python 3 user to be able to `import >> garlicsim` and refer to the module all the time as `garlicsim`, but that it >> will really be `garlicsim_py3`. >> > > Why not just develop them both with the same package name? Your project > name (in setup(name='...')) can still be different. > > Oh, I didn't even know that was possible. That sounds great, and I wish I would have known about this option 6 months ago. If this works well, I'll start converting my entire codebase to this method. Before I jump in, is there anything important I should keep in mind, any possible problems? Also, this is plain setuptools, right? I mean, not requiring Distribute. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu May 27 23:28:27 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 27 May 2010 17:28:27 -0400 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: <20100527210334.DCAE73A409D@sparrow.telecommunity.com> Message-ID: <20100527212826.0146B3A409D@sparrow.telecommunity.com> At 11:16 PM 5/27/2010 +0200, cool-RR wrote: >On Thu, May 27, 2010 at 11:03 PM, P.J. Eby ><pje at telecommunity.com> wrote: >At 10:40 PM 5/27/2010 +0200, cool-RR wrote: >Is there a way that `garlicsim_py3` will install itself under the >alias `garlicsim`? What I want is for a Python 3 user to be able to >`import garlicsim` and refer to the module all the time as >`garlicsim`, but that it will really be `garlicsim_py3`. > > >Why not just develop them both with the same package name? Your >project name (in setup(name='...')) can still be different. > > >Oh, I didn't even know that was possible. That sounds great, and I >wish I would have known about this option 6 months ago. > >If this works well, I'll start converting my entire codebase to this >method. Before I jump in, is there anything important I should keep >in mind, any possible problems? > >Also, this is plain setuptools, right? I mean, not requiring Distribute. It works with plain distutils, too. Just use setup(name='garlicsym_py3', packages=['garlicsym']) in your Python 3 branch of setup.py, and setup(name='garlicsym', packages=['garlicsym']) in your Python 2 branch. From cournape at gmail.com Fri May 28 00:25:31 2010 From: cournape at gmail.com (David Cournapeau) Date: Fri, 28 May 2010 07:25:31 +0900 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: Message-ID: On Fri, May 28, 2010 at 5:40 AM, cool-RR wrote: > Hello all! > I want to ask you a question regarding something I want to do with my app. > The reason I'm asking here is because I understand that Distribute somehow > installs itself as "setuptools". (I've never looked into the details of it.) Your choice, but please do not do this. I really hate the behavior of distribute in that regard, and it has already wasted several hours of my time. > And I want something similar, so maybe you can advise me how to do it. > I'm developing a package called `garlicsim`. The package is intended for > Python 2.X, but I am also offerring Python 3 support on a different fork > called `garlicsim_py3`.(1) > So both of these packages live side by side on PyPI, and Python 3 users > install `garlicsim_py3`, and Python 2 users install `garlicsim`. > The problem is: When third party modules want to use garlicsim, they should > have one package name to refer to, not two. Sure, they can do something like > this: > ?? ?try: > ?? ? ? ?import garlicsim > ?? ?except ImportError: > ?? ? ? ?import garlicsim_py3 as garlicsim > But I would prefer not to make the developers of these modules do this. That's so much better than having some magic there. There are other solutions to your problem, but please, do not play with names like that. That's really not necessary. David From cool-rr at cool-rr.com Fri May 28 01:11:49 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Fri, 28 May 2010 01:11:49 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: Message-ID: On Fri, May 28, 2010 at 12:25 AM, David Cournapeau wrote: > On Fri, May 28, 2010 at 5:40 AM, cool-RR wrote: > > Hello all! > > I want to ask you a question regarding something I want to do with my > app. > > The reason I'm asking here is because I understand that Distribute > somehow > > installs itself as "setuptools". (I've never looked into the details of > it.) > > Your choice, but please do not do this. I really hate the behavior of > distribute in that regard, and it has already wasted several hours of > my time. > > > And I want something similar, so maybe you can advise me how to do it. > > I'm developing a package called `garlicsim`. The package is intended for > > Python 2.X, but I am also offerring Python 3 support on a different fork > > called `garlicsim_py3`.(1) > > So both of these packages live side by side on PyPI, and Python 3 users > > install `garlicsim_py3`, and Python 2 users install `garlicsim`. > > The problem is: When third party modules want to use garlicsim, they > should > > have one package name to refer to, not two. Sure, they can do something > like > > this: > > try: > > import garlicsim > > except ImportError: > > import garlicsim_py3 as garlicsim > > But I would prefer not to make the developers of these modules do this. > > That's so much better than having some magic there. There are other > solutions to your problem, but please, do not play with names like > that. That's really not necessary. > > David > Thanks for the advice David. If P.J.'s suggestion will work for me, I wouldn't have to take that route. The thing I want the most right now is anyone's opinion of what could possibly go wrong with the scheme that P.J. offered, before I start moving my codebase to it. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Fri May 28 02:36:42 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 27 May 2010 20:36:42 -0400 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: Message-ID: <20100528003642.AD2243A409D@sparrow.telecommunity.com> At 01:11 AM 5/28/2010 +0200, cool-RR wrote: >The thing I want the most right now is anyone's opinion of what >could possibly go wrong with the scheme that P.J. offered, before I >start moving my codebase to it. It's quite routine. Certainly, few of my PyPI projects have the same project and package name. For example, 'Importing' contains the module 'peak.util.imports'. It's also done by many other packages I'm not involved with. For example, 'AeroCalc' is the PyPI name of a project that ships an 'aerocalc' package, 'AdjectorClient' ships various packages such as adjector.core and adjector.model... and those are just two things I found by randomly clicking on the A's in the giant list of PyPI packages! I'm sure there will be some found in every letter of the alphabet. ;-) The only thing that can go wrong is if somebody installs the Python 2 and Python 3 versions to the same installation location. However, as long as both versions alert the user when used on the wrong Python version, they will quickly discover their mistake. ;-) From cool-rr at cool-rr.com Fri May 28 02:53:05 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Fri, 28 May 2010 02:53:05 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: <20100528003642.AD2243A409D@sparrow.telecommunity.com> References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> Message-ID: On Fri, May 28, 2010 at 2:36 AM, P.J. Eby wrote: > At 01:11 AM 5/28/2010 +0200, cool-RR wrote: > >> The thing I want the most right now is anyone's opinion of what could >> possibly go wrong with the scheme that P.J. offered, before I start moving >> my codebase to it. >> > > It's quite routine. Certainly, few of my PyPI projects have the same > project and package name. For example, 'Importing' contains the module > 'peak.util.imports'. > > It's also done by many other packages I'm not involved with. For example, > 'AeroCalc' is the PyPI name of a project that ships an 'aerocalc' package, > 'AdjectorClient' ships various packages such as adjector.core and > adjector.model... and those are just two things I found by randomly > clicking on the A's in the giant list of PyPI packages! > > I'm sure there will be some found in every letter of the alphabet. ;-) > > The only thing that can go wrong is if somebody installs the Python 2 and > Python 3 versions to the same installation location. > However, as long as both versions alert the user when used on the wrong > Python version, they will quickly discover their mistake. ;-) > Yes, I thought about this too. You suggest doing that on the setup script, the package itself, or both? To anyone else listening: If you think of ANYTHING that can go wrong with this scheme, please speak up. I really like knowing about possible problems in advance. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri May 28 02:59:23 2010 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 27 May 2010 20:59:23 -0400 Subject: [Distutils] pkginfo python 3 port In-Reply-To: <20100527171044.7c218718@heresy> References: <3702344C-48D9-4F9F-8C21-7CECEFA21466@activestate.com> <4BFD8CF0.1090201@agendaless.com> <30684B58-A48E-47B2-BBFC-C9FF0B2EB2BA@activestate.com> <4BFEA679.2010103@agendaless.com> <03B67BBD-832E-4253-93CD-D1AA0DA3A442@activestate.com> <20100527171044.7c218718@heresy> Message-ID: <4BFF156B.7010200@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > On May 27, 2010, at 10:25 AM, Sridhar Ratnakumar wrote: > >> Is there a way to parse a RFC 822 message in Python 3? > > If it's ASCII, you should have no problems using email.parser.Parser. The issue is that its behavior is subtly different from the now-removed rfc822 parser:: $ /opt/Python-2.6.5/bin/python Python 2.6.5 (r265:79063, Apr 6 2010, 14:45:18) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from StringIO import StringIO >>> with_multiline = StringIO("""\ ... Description: this is a multiline RFC 822 ... header.""") >>> from rfc822 import Message >>> rfc_msg = Message(with_multiline) >>> with_multiline.seek(0) >>> from email.parser import Parser >>> email_msg = Parser().parse(with_multiline) >>> rfc_msg.getheader('Description') 'this is a multiline RFC 822\n header.' >>> email_msg.get('Description') 'this is a multiline RFC 822\n header.' Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv/FWsACgkQ+gerLs4ltQ67SwCeIWLyFj2c7rLb/cpcoxZ4sUzF eHYAoInVb2cDZsIwOB0loSkZ3d9gAiDi =Q1BW -----END PGP SIGNATURE----- From ktenney at gmail.com Fri May 28 04:12:57 2010 From: ktenney at gmail.com (Kent Tenney) Date: Thu, 27 May 2010 21:12:57 -0500 Subject: [Distutils] Unexpected error Message-ID: In a docutils svn checkout. [docutils/trunk/docutils]$ python setup.py install --root /tmp OK [docutils/trunk/docutils]$ python setup.py install_data --root /tmp distutils.errors.DistutilsFileError: could not delete '/usr/local/lib/python2.6/dist-packages/docutils/parsers/rst/include/README.txt': Permission denied What am I missing here? Thanks, Kent From cournape at gmail.com Fri May 28 08:16:31 2010 From: cournape at gmail.com (David Cournapeau) Date: Fri, 28 May 2010 15:16:31 +0900 Subject: [Distutils] Unexpected error In-Reply-To: References: Message-ID: On Fri, May 28, 2010 at 11:12 AM, Kent Tenney wrote: > In a docutils svn checkout. > > [docutils/trunk/docutils]$ python setup.py install --root /tmp > OK > > [docutils/trunk/docutils]$ python setup.py install_data --root /tmp > > distutils.errors.DistutilsFileError: > could not delete > '/usr/local/lib/python2.6/dist-packages/docutils/parsers/rst/include/README.txt': > Permission denied Well, do you have writing permision in that directory ? Generally, you need admin priviledges to write anything in /usr/local, and tmp is writable by anyone, David From destrero at imavis.com Thu May 27 10:54:27 2010 From: destrero at imavis.com (Augusto Destrero) Date: Thu, 27 May 2010 10:54:27 +0200 Subject: [Distutils] "Out of source" bdist_egg, missing metadata Message-ID: <201005271054.27983.destrero@imavis.com> Hi, I've a question on setuptools egg creation. I've a directory structure like this: python_libs/ |---- setup_foo | |---- setup.py |---- src | |---- foo | | |---- __init__.py | | |---- foo_module.py | |---- bar | | |---- __init__.py | | |---- bar_module.py | |---- common | | |---- __init__.py | | |---- common_module.py Basically I have a src directory containing three packages (foo, bar and common), and I have a setup_foo directory OUTSIDE src, where I want to build an egg containing foo and common packages, and NOT bar package. In the setup_foo/setup.py script I have something like this: # -*- coding: utf-8 -*- from setuptools import setup, find_packages src_dir = "../src" setup(name='foo', version='0.1', description="foo and common packages", packages=find_packages(src_dir, exclude=['bar', 'bar.*']), package_dir={'': src_dir}, zip_safe=False, install_requires=[ 'PasteDeploy', ], entry_points=""" [paste.app_factory] foo_app = foo.foo_module:app_factory common_app = common.common_module:app_factory """, ) You can see that I have two entry points (in this case for use with PasteDeploy in a wsgi project). If I build an egg with that file: $ cd setup_foo/ $ python setup.py bdist_egg I can find a foo.egg-info directory under src which contains all metadata: $ ls src/foo.egg-info/ dependency_links.txt entry_points.txt not-zip-safe PKG-INFO requires.txt SOURCES.txt top_level.txt But in the egg created under setup_foo/dist I don't have those metadata: $ cd setup_foo/dist/ $ unzip foo-0.1-py2.6.egg $ ls EGG-INFO/ not-zip-safe SOURCES.txt And of course, after installing that egg with easy_install, PasteDeploy can't find the entry points. Is this a setuptools bug or I'm doing something wrong? I use setuptools 0.6.10 distributed in Ubuntu Lucid. Thank you in advance for your help, and excuse me for the lenghty email! Please answer also to my email address, because I'm not a list subscriber. -- Augusto Destrero From tseaver at palladion.com Fri May 28 18:18:38 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 28 May 2010 12:18:38 -0400 Subject: [Distutils] Unexpected error In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Cournapeau wrote: > On Fri, May 28, 2010 at 11:12 AM, Kent Tenney wrote: >> In a docutils svn checkout. >> >> [docutils/trunk/docutils]$ python setup.py install --root /tmp >> OK >> >> [docutils/trunk/docutils]$ python setup.py install_data --root /tmp >> >> distutils.errors.DistutilsFileError: >> could not delete >> '/usr/local/lib/python2.6/dist-packages/docutils/parsers/rst/include/README.txt': >> Permission denied > > Well, do you have writing permision in that directory ? Generally, you > need admin priviledges to write anything in /usr/local, and tmp is > writable by anyone, The issue is that distutils is ignoring the '--root' passed to 'install_data'. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEUEARECAAYFAkv/7NgACgkQ+gerLs4ltQ7MxACVEcSHRW5y5doD2765QSrGhnSK ogCfU6Impyk5lGSGKamRTbY5Td7Tb3c= =7YDb -----END PGP SIGNATURE----- From ktenney at gmail.com Fri May 28 19:07:55 2010 From: ktenney at gmail.com (Kent Tenney) Date: Fri, 28 May 2010 12:07:55 -0500 Subject: [Distutils] Unexpected error In-Reply-To: References: Message-ID: On Fri, May 28, 2010 at 11:18 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > David Cournapeau wrote: >> On Fri, May 28, 2010 at 11:12 AM, Kent Tenney wrote: >>> In a docutils svn checkout. >>> >>> [docutils/trunk/docutils]$ python setup.py install --root /tmp >>> OK >>> >>> [docutils/trunk/docutils]$ python setup.py install_data --root /tmp >>> >>> distutils.errors.DistutilsFileError: >>> could not delete >>> '/usr/local/lib/python2.6/dist-packages/docutils/parsers/rst/include/README.txt': >>> Permission denied >> >> Well, do you have writing permision in that directory ? Generally, you >> need admin priviledges to write anything in ?/usr/local, and tmp is >> writable by anyone, > > The issue is that distutils is ignoring the '--root' passed to > 'install_data'. Right, I should have mentioned that explicitly. [docutils/trunk/docutils]$ python setup.py install_data --help ... Options for 'smart_install_data' command: --install-dir (-d) base directory for installing data files (default: installation base dir) --root install everything relative to this alternate root directory --force (-f) force installation (overwrite existing files) ... > > > Tres. > - -- > =================================================================== > Tres Seaver ? ? ? ? ?+1 540-429-0999 ? ? ? ? ?tseaver at palladion.com > Palladion Software ? "Excellence by Design" ? ?http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEUEARECAAYFAkv/7NgACgkQ+gerLs4ltQ7MxACVEcSHRW5y5doD2765QSrGhnSK > ogCfU6Impyk5lGSGKamRTbY5Td7Tb3c= > =7YDb > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From barry at python.org Fri May 28 19:35:54 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 28 May 2010 13:35:54 -0400 Subject: [Distutils] pkginfo python 3 port In-Reply-To: <4BFF156B.7010200@palladion.com> References: <3702344C-48D9-4F9F-8C21-7CECEFA21466@activestate.com> <4BFD8CF0.1090201@agendaless.com> <30684B58-A48E-47B2-BBFC-C9FF0B2EB2BA@activestate.com> <4BFEA679.2010103@agendaless.com> <03B67BBD-832E-4253-93CD-D1AA0DA3A442@activestate.com> <20100527171044.7c218718@heresy> <4BFF156B.7010200@palladion.com> Message-ID: <20100528133554.52eb5e3d@heresy> On May 27, 2010, at 08:59 PM, Tres Seaver wrote: >Barry Warsaw wrote: >> On May 27, 2010, at 10:25 AM, Sridhar Ratnakumar wrote: >> >>> Is there a way to parse a RFC 822 message in Python 3? >> >> If it's ASCII, you should have no problems using email.parser.Parser. > >The issue is that its behavior is subtly different from the now-removed >rfc822 parser:: > > $ /opt/Python-2.6.5/bin/python > Python 2.6.5 (r265:79063, Apr 6 2010, 14:45:18) > [GCC 4.3.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from StringIO import StringIO > >>> with_multiline = StringIO("""\ > ... Description: this is a multiline RFC 822 > ... header.""") > >>> from rfc822 import Message > >>> rfc_msg = Message(with_multiline) > >>> with_multiline.seek(0) > >>> from email.parser import Parser > >>> email_msg = Parser().parse(with_multiline) > >>> rfc_msg.getheader('Description') > 'this is a multiline RFC 822\n header.' > >>> email_msg.get('Description') > 'this is a multiline RFC 822\n header.' If I'm reading this correctly, the "problem" is that rfc822 collapses continuation whitespace and email.parser preserves it? Isn't the email package (more) correct, and what specific problem does that cause? Or is it just that it's different so tools have to catch up to that? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From pje at telecommunity.com Fri May 28 19:44:18 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 28 May 2010 13:44:18 -0400 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> Message-ID: <20100528174419.A33F83A4119@sparrow.telecommunity.com> At 02:53 AM 5/28/2010 +0200, cool-RR wrote: >On Fri, May 28, 2010 at 2:36 AM, P.J. Eby ><pje at telecommunity.com> wrote: >However, as long as both versions alert the user when used on the >wrong Python version, they will quickly discover their mistake. ;-) > >Yes, I thought about this too. You suggest doing that on the setup >script, the package itself, or both? The package itself; doing it in the setup script is ridiculously hard. (Not detecting the Python version, but detecting whether you're installing both versions to the same location.) From tseaver at palladion.com Fri May 28 19:54:35 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 28 May 2010 13:54:35 -0400 Subject: [Distutils] pkginfo python 3 port In-Reply-To: <20100528133554.52eb5e3d@heresy> References: <3702344C-48D9-4F9F-8C21-7CECEFA21466@activestate.com> <4BFD8CF0.1090201@agendaless.com> <30684B58-A48E-47B2-BBFC-C9FF0B2EB2BA@activestate.com> <4BFEA679.2010103@agendaless.com> <03B67BBD-832E-4253-93CD-D1AA0DA3A442@activestate.com> <20100527171044.7c218718@heresy> <4BFF156B.7010200@palladion.com> <20100528133554.52eb5e3d@heresy> Message-ID: <4C00035B.9010603@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > On May 27, 2010, at 08:59 PM, Tres Seaver wrote: > >> Barry Warsaw wrote: >>> On May 27, 2010, at 10:25 AM, Sridhar Ratnakumar wrote: >>> >>>> Is there a way to parse a RFC 822 message in Python 3? >>> If it's ASCII, you should have no problems using email.parser.Parser. >> The issue is that its behavior is subtly different from the now-removed >> rfc822 parser:: >> >> $ /opt/Python-2.6.5/bin/python >> Python 2.6.5 (r265:79063, Apr 6 2010, 14:45:18) >> [GCC 4.3.3] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> from StringIO import StringIO >> >>> with_multiline = StringIO("""\ >> ... Description: this is a multiline RFC 822 >> ... header.""") >> >>> from rfc822 import Message >> >>> rfc_msg = Message(with_multiline) >> >>> with_multiline.seek(0) >> >>> from email.parser import Parser >> >>> email_msg = Parser().parse(with_multiline) >> >>> rfc_msg.getheader('Description') >> 'this is a multiline RFC 822\n header.' >> >>> email_msg.get('Description') >> 'this is a multiline RFC 822\n header.' > > If I'm reading this correctly, the "problem" is that rfc822 collapses > continuation whitespace and email.parser preserves it? Isn't the email > package (more) correct, "More correct" is debateable: The email.parser module does not remove the newline, for instance, which is what RFC2822 suggests for "unfolding" header lines: http://www.faqs.org/rfcs/rfc2822.html Collapsing extra leading whitespace in header continuation lines seems like a reasonable strategy: lines created by "folding" per RFC (2)822 won't normally have them, while those which do (e.g, as created by distutils, or perhaps by hand) do, but they aren't meaningful. > and what specific problem does that cause? Or is it > just that it's different so tools have to catch up to that? In particular, pkginfo wants to run across a wide range of Python versions, with Python 2.4 still actively supported. I therefore need to fall back to the rfc822 module when the newer module is not present. I have chosen for the moment to enforce the collapsing where email.parser is used. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkwAA1YACgkQ+gerLs4ltQ6ZRwCeOuwr3bq/h6BJqWWNfUB+qygB /fQAoI5daS3qA/yYiEF6s+PsDaGPcxOn =KN6V -----END PGP SIGNATURE----- From cool-rr at cool-rr.com Fri May 28 20:00:16 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Fri, 28 May 2010 20:00:16 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: <20100528174419.A33F83A4119@sparrow.telecommunity.com> References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> <20100528174419.A33F83A4119@sparrow.telecommunity.com> Message-ID: On Fri, May 28, 2010 at 7:44 PM, P.J. Eby wrote: > At 02:53 AM 5/28/2010 +0200, cool-RR wrote: > > On Fri, May 28, 2010 at 2:36 AM, P.J. Eby < >> pje at telecommunity.com> wrote: >> However, as long as both versions alert the user when used on the wrong >> Python version, they will quickly discover their mistake. ;-) >> >> Yes, I thought about this too. You suggest doing that on the setup script, >> the package itself, or both? >> > > The package itself; doing it in the setup script is ridiculously hard. > (Not detecting the Python version, but detecting whether you're installing > both versions to the same location.) > I'm not sure we're on the same page here. My intention is that Python 2.x users will use only `garlicsim`, and Python 3.x users will use only `garlicsim_py3`. Why would I want to detect anything beyond the Python version in the setup script? Ram., Ram Rachum -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri May 28 20:02:18 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 28 May 2010 14:02:18 -0400 Subject: [Distutils] pkginfo python 3 port In-Reply-To: <4C00035B.9010603@palladion.com> References: <3702344C-48D9-4F9F-8C21-7CECEFA21466@activestate.com> <4BFD8CF0.1090201@agendaless.com> <30684B58-A48E-47B2-BBFC-C9FF0B2EB2BA@activestate.com> <4BFEA679.2010103@agendaless.com> <03B67BBD-832E-4253-93CD-D1AA0DA3A442@activestate.com> <20100527171044.7c218718@heresy> <4BFF156B.7010200@palladion.com> <20100528133554.52eb5e3d@heresy> <4C00035B.9010603@palladion.com> Message-ID: <20100528140218.3cec1afc@heresy> On May 28, 2010, at 01:54 PM, Tres Seaver wrote: >"More correct" is debateable: The email.parser module does not remove >the newline, for instance, which is what RFC2822 suggests for >"unfolding" header lines: > > http://www.faqs.org/rfcs/rfc2822.html > >Collapsing extra leading whitespace in header continuation lines seems >like a reasonable strategy: lines created by "folding" per RFC (2)822 >won't normally have them, while those which do (e.g, as created by >distutils, or perhaps by hand) do, but they aren't meaningful. Right. Over in email-sig land we're talking about how to access both the raw header (i.e. what you parsed) and the intended semantic header which would be the unfolded value. No need to re-hash that here in distutils-sig. >> and what specific problem does that cause? Or is it >> just that it's different so tools have to catch up to that? > >In particular, pkginfo wants to run across a wide range of Python >versions, with Python 2.4 still actively supported. I therefore need to >fall back to the rfc822 module when the newer module is not present. I >have chosen for the moment to enforce the collapsing where email.parser >is used. Probably best to manually unfold the header value regardless of where you got it from. The email package should (and hopefully someday will) make this easier. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From regebro at gmail.com Fri May 28 20:12:51 2010 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 28 May 2010 20:12:51 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> <20100528174419.A33F83A4119@sparrow.telecommunity.com> Message-ID: On Fri, May 28, 2010 at 20:00, cool-RR wrote: > I'm not sure we're on the same page here. My intention is that Python 2.x > users will use only `garlicsim`, and Python 3.x users will use only > `garlicsim_py3`. Why would you want that? Reasonably it should be called garlicsim in both cases. If not you make it needlessly painful to move from Python 2 to Python 3. Also, even though you want your code to be clean, the amount of compatibility hacks to support both Python 2 and Python 3 you would need is probably not very large if you use Distribute and it's 2to3 support. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cool-rr at cool-rr.com Fri May 28 20:17:42 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Fri, 28 May 2010 20:17:42 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> <20100528174419.A33F83A4119@sparrow.telecommunity.com> Message-ID: On Fri, May 28, 2010 at 8:12 PM, Lennart Regebro wrote: > On Fri, May 28, 2010 at 20:00, cool-RR wrote: > > I'm not sure we're on the same page here. My intention is that Python 2.x > > users will use only `garlicsim`, and Python 3.x users will use only > > `garlicsim_py3`. > > Why would you want that? Reasonably it should be called garlicsim in both > cases. > If not you make it needlessly painful to move from Python 2 to Python 3. > As P.J. said, the package will have the same name `garlicsim` in both versions, but the name on PyPI of the Python 3 version will be `garlicsim_py3`. > Also, even though you want your code to be clean, the amount of > compatibility hacks to support both Python 2 and Python 3 you would > need is probably not very large if you use Distribute and it's 2to3 > support. > Funny you mention it just now. Python 3 is just giving me an obscure problem that I (a) don't know how to solve and (b) seriously doubt any 2to3 algorithm will handle. Here it is: http://stackoverflow.com/questions/2930792/pickling-an-unbound-method-in-python-3 Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Fri May 28 20:23:40 2010 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 28 May 2010 20:23:40 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> <20100528174419.A33F83A4119@sparrow.telecommunity.com> Message-ID: On Fri, May 28, 2010 at 20:17, cool-RR wrote: > As P.J. said, the package will have the same name `garlicsim` in both > versions, but the name on PyPI of the Python 3 version will be > `garlicsim_py3`. Ah, OK, I misunderstood there, i thought you meant the other way around. > Funny you mention it just now. Python 3 is just giving me an obscure problem > that I (a) don't know how to solve and (b) seriously doubt any 2to3 > algorithm will handle. If has nothing to do with conversion at all. If you can solve it (which I hope you can't) it's going to work 2to3 or no 2to3. But hopefully you just find a way to *not* pickle methods. :-) From cool-rr at cool-rr.com Fri May 28 20:31:13 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Fri, 28 May 2010 20:31:13 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> <20100528174419.A33F83A4119@sparrow.telecommunity.com> Message-ID: On Fri, May 28, 2010 at 8:23 PM, Lennart Regebro wrote: > On Fri, May 28, 2010 at 20:17, cool-RR wrote: > > As P.J. said, the package will have the same name `garlicsim` in both > > versions, but the name on PyPI of the Python 3 version will be > > `garlicsim_py3`. > > Ah, OK, I misunderstood there, i thought you meant the other way around. > > > Funny you mention it just now. Python 3 is just giving me an obscure > problem > > that I (a) don't know how to solve and (b) seriously doubt any 2to3 > > algorithm will handle. > > If has nothing to do with conversion at all. If you can solve it > (which I hope you can't) it's going to work 2to3 or no 2to3. But > hopefully you just find a way to *not* pickle methods. :-) > That's easy to say when it's someone else's problem and not yours. I know that pickling methods sounds wrong. If you'll want details on why I want to pickle modules I'll be happy to share, though probably off-list as it will probably not be very interesting to the members of this list. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Fri May 28 21:14:26 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 28 May 2010 15:14:26 -0400 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> <20100528174419.A33F83A4119@sparrow.telecommunity.com> Message-ID: <20100528191442.646DC3A4119@sparrow.telecommunity.com> At 08:00 PM 5/28/2010 +0200, cool-RR wrote: >On Fri, May 28, 2010 at 7:44 PM, P.J. Eby ><pje at telecommunity.com> wrote: >At 02:53 AM 5/28/2010 +0200, cool-RR wrote: > >On Fri, May 28, 2010 at 2:36 AM, P.J. Eby ><pje at telecommunity.com> >wrote: >However, as long as both versions alert the user when used on the >wrong Python version, they will quickly discover their mistake. ;-) > >Yes, I thought about this too. You suggest doing that on the setup >script, the package itself, or both? > > >The package itself; doing it in the setup script is ridiculously >hard. (Not detecting the Python version, but detecting whether >you're installing both versions to the same location.) > > >I'm not sure we're on the same page here. My intention is that >Python 2.x users will use only `garlicsim`, and Python 3.x users >will use only `garlicsim_py3`. But both projects will contain a package named 'garlicsim' - so if they're installed to the same location, then either one will overwrite the other (distutils/pip install) or override the other (setuptools). In principle, they won't be installed to the same location, unless somebody has a directory that's used for both Python 2 packages and Python 3 packages -- and if they do that, they'll have other problems. (i.e., they shouldn't be doing it) > Why would I want to detect anything beyond the Python version in > the setup script? Even though you detect the version in the setup script, it won't stop the problem of installing both versions to the same location. That's why the package itself needs to give a good error message if imported from the wrong Python version, to handle the case where somebody's sys.path is wrong (e.g. due to a PYTHONPATH that includes both Python 2 and 3 packages). Again, people are going to have other problems if they do that, unrelated to your package. Putting in the check is just a courtesy measure so they don't have to spend a lot of time figuring out what they did wrong. From cool-rr at cool-rr.com Fri May 28 21:49:10 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Fri, 28 May 2010 21:49:10 +0200 Subject: [Distutils] Having a Python package install itself under a different name In-Reply-To: <20100528191442.646DC3A4119@sparrow.telecommunity.com> References: <20100528003642.AD2243A409D@sparrow.telecommunity.com> <20100528174419.A33F83A4119@sparrow.telecommunity.com> <20100528191442.646DC3A4119@sparrow.telecommunity.com> Message-ID: On Fri, May 28, 2010 at 9:14 PM, P.J. Eby wrote: > At 08:00 PM 5/28/2010 +0200, cool-RR wrote: > >> On Fri, May 28, 2010 at 7:44 PM, P.J. Eby < >> pje at telecommunity.com> wrote: >> At 02:53 AM 5/28/2010 +0200, cool-RR wrote: >> >> On Fri, May 28, 2010 at 2:36 AM, P.J. Eby <> >pje at telecommunity.com> wrote: >> However, as long as both versions alert the user when used on the wrong >> Python version, they will quickly discover their mistake. ;-) >> >> Yes, I thought about this too. You suggest doing that on the setup script, >> the package itself, or both? >> >> >> The package itself; doing it in the setup script is ridiculously hard. >> (Not detecting the Python version, but detecting whether you're installing >> both versions to the same location.) >> >> >> I'm not sure we're on the same page here. My intention is that Python 2.x >> users will use only `garlicsim`, and Python 3.x users will use only >> `garlicsim_py3`. >> > > But both projects will contain a package named 'garlicsim' - so if they're > installed to the same location, then either one will overwrite the other > (distutils/pip install) or override the other (setuptools). > > In principle, they won't be installed to the same location, unless somebody > has a directory that's used for both Python 2 packages and Python 3 packages > -- and if they do that, they'll have other problems. (i.e., they shouldn't > be doing it) > > > > Why would I want to detect anything beyond the Python version in the >> setup script? >> > > Even though you detect the version in the setup script, it won't stop the > problem of installing both versions to the same location. That's why the > package itself needs to give a good error message if imported from the wrong > Python version, to handle the case where somebody's sys.path is wrong (e.g. > due to a PYTHONPATH that includes both Python 2 and 3 packages). > > Again, people are going to have other problems if they do that, unrelated > to your package. Putting in the check is just a courtesy measure so they > don't have to spend a lot of time figuring out what they did wrong. > Okay. Thanks for the tips, I'll put version checks in both the setup scripts and the packages. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Sat May 29 01:13:41 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 29 May 2010 01:13:41 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 Message-ID: Hello, Distutils2 is going to be added back in Python (hopefully in 3.2) and without an install script, it's pretty useless as-is. We've discussed during the summit at Pycon to create some kind of bootstrap script in Python, to allow people to set up an installer of their choice, but I think it's a bad idea. = summary of the summit proposal = IIRC: - an "install" script is added in the scripts, for people to install distributions in Python. Could be called "pyinstall" for example. - when first used, it would ask the user to choose a third-party installer (like Pip). Then it would download it and install it with a simple "python setup.py install" - from there, the install script would be linked to that installer. If I recall it correctly, this feature was proposed to be able to have a "modern" installer in Python without including it in the standard library. (so it can have its own shorter release cycles). The bootstrap story would just make it easier for people to get an installer, without having to do extra manual steps. The problem I have with this approach is that we need to manage somewhere at PyPI a list of potential installers, and maybe deal with upgrades and replacements. Plus, I am not sure that a user will really understand what to do when he's asked to chose an installer. Sounds like something we should only ask to power users, and people that know what they are doing with p7g. So a bootstrap script is useless for them. = alternative proposal = Let's add that script but powered by Distutils2. It could be Pip if people from this project think it's a good idea and want to merge, or an easy_install derivation, or a new script from scratch. IOW: you get an installer for free in the stdlib without having to think. Now for the problem of the release cycle (e.g. once in the standard library it has to wait 18 months for a new version), I propose that Distutils2 allow the usage of a third party installer through configuration. IOW, Distutils2 would ship with an installer, but could use through a simple change in distutils.cfg, another one installed by a third party project that is more recent. For this to work, we can define an installer standard interface ala wsgi. Basically, we state that an installer has to implement a simple function that takes a name of a project to install, and an optional version predicate: def install(name, version=None): # if version is None, it means the latest one. ... This needs more work, uninstall is missing in that description, and what about the script options, etc. but you get the idea: make sure people can use the installer of their choice, if it turns out that the one provided by Distutils2 is not good enough anymore for any reason. Any opinion ? Regards, Tarek -- Tarek Ziad? | http://ziade.org From pje at telecommunity.com Sat May 29 02:28:14 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 28 May 2010 20:28:14 -0400 Subject: [Distutils] pip vs easy_install vs distutils2 Message-ID: <20100529002815.B5DFE3A4119@sparrow.telecommunity.com> At 01:13 AM 5/29/2010 +0200, Tarek Ziad? wrote: >The problem I have with this approach is that we need to manage >somewhere at PyPI a list of potential installers, >and maybe deal with upgrades and replacements. Plus, I am not sure >that a user will really understand what to >do when he's asked to chose an installer. Sounds like something we >should only ask to power users, and >people that know what they are doing with p7g. So a bootstrap script >is useless for them. Actually, the way it would (presumably) work would be just a script like Guido's that downloads a source archive from PyPI and runs setup.py install. So, it's not really a matter of "choosing" an installer -- it'd just be, "download and install a package from source". If the package itself needs one of those other things in order to build itself or install dependencies, then the next stage of bootstrap is its problem (e.g. via ez_setup), not the stdlib's. So, I don't see the "power users only" objection making sense, at least if we drop back to what I believe was Guido's original proposal for a bootstrap script a few years ago. From ziade.tarek at gmail.com Sat May 29 10:19:34 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 29 May 2010 10:19:34 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> Message-ID: 2010/5/29 P.J. Eby : > At 01:13 AM 5/29/2010 +0200, Tarek Ziad? wrote: >> >> The problem I have with this approach is that we need to manage >> somewhere at PyPI a list of potential installers, >> and maybe deal with upgrades and replacements. Plus, I am not sure >> that a user will really understand what to >> do when he's asked to chose an installer. Sounds like something we >> should only ask to power users, and >> people that know what they are doing with p7g. So a bootstrap script >> is useless for them. > > Actually, the way it would (presumably) work would be just a script like > Guido's that downloads a source archive from PyPI and runs setup.py install. > ?So, it's not really a matter of "choosing" an installer -- it'd just be, > "download and install a package from source". > > If the package itself needs one of those other things in order to build > itself or install dependencies, then the next stage of bootstrap is its > problem (e.g. via ez_setup), not the stdlib's. > > So, I don't see the "power users only" objection making sense, at least if > we drop back to what I believe was Guido's original proposal for a bootstrap > script a few years ago. I was not thinking about this proposal. If this what Guido proposed at the summit, then I misunderstood. I was thiking about a bootstrap process on the end-user side, to set up an installer once for all, on a fresh Python. The problem with the feature you describe (bootstrap embed in the archive itself) is that we imply that the packager chooses himself an installer and forces the end-user to use it when he installs the software. This means that the end user might end up with several installers installed for the same Python. It also implies that the developer provides a smart setup.py script that embeds that bootstrap, and runs some code. I think separating the concerns and letting the end user pick/use explicitly *one* installer globally is better because several installers won't compete on the target system (even if we supposely want them all to be compatible in the future). Of course, this is only true for source distributions. It will also allow distributions to be "dumb" envelopes with static metadata that are the same all the time, no matter which tool created them, and eventually remove setup.py in favor of statically described metadata using PEP 345. Today, for instance, if an installer wants to install a distribution based on setuptools, it has to run the "egg_info" command to extract the metadata, on the target system. Being able to get those metadata without running any code would be better. For instance, installers could list all dependencies for a project by querying PyPI with zero download/execution. (thanks to PEP 345 environment markers) What would make more sense I think, would be to have all installers an identical archive for a given project, that doesn't need more code to be run to get all the metadata. So at the end, the end user would chose an installer that is compatible with these archive, and know how to install them. In other words, have ez_setup for example, run once for all at the Python level, and be THE installer. Or run a pip_setup or whatever. Regards, Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Sat May 29 10:25:26 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 29 May 2010 10:25:26 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> Message-ID: On Sat, May 29, 2010 at 10:19 AM, Tarek Ziad? wrote: [..] > > It will also allow distributions to be "dumb" envelopes with static > metadata that are the same all the time, no matter which tool created > them, and eventually remove setup.py in favor of statically described > metadata using PEP 345. + statically described distribution content From pje at telecommunity.com Sat May 29 18:32:17 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 29 May 2010 12:32:17 -0400 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> Message-ID: <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> At 10:19 AM 5/29/2010 +0200, Tarek Ziad? wrote: >I was not thinking about this proposal. If this what Guido proposed at >the summit, then I misunderstood. I don't know what he proposed at the summit - I'm referring to the bootstrap script he wrote that actually does this. It was a couple years ago on Python-Dev, IIRC. >The problem with the feature you describe (bootstrap embed in the >archive itself) is that we imply that the packager chooses himself an >installer and forces the end-user to use it when he installs the >software. This means that the end user might end up with several >installers installed for the same Python. So? It's not like they're going to accidentally run 'easy_install' when they meant to type 'pip', or vice versa. ;-) >It also implies that the >developer provides a smart setup.py script that embeds that bootstrap, >and runs some code. Only if they need something beyond what the distutils provide. >It will also allow distributions to be "dumb" envelopes with static >metadata that are the same all the time, no matter which tool created >them, and eventually remove setup.py in favor of statically described >metadata using PEP 345. For packages with complex build requirements or distutils extensions (e.g. numpy), this is unlikely to happen any time soon. Conversely, for packages where this *is* the case, the current distutils is adequate, and having a bootstrapper that can install them would be a win. >Today, for instance, if an installer wants to install a distribution >based on setuptools, it has to run the "egg_info" command to extract >the metadata, on the target system. It's a good thing Guido loaned me the time machine so I could go back to 2005 and fix that one, then: http://svn.python.org/view/sandbox/trunk/setuptools/setuptools/command/sdist.py?view=diff&r1=41093&r2=41094 Five years should be long enough ago that by now all the setuptools-based source distributions on PyPI today will already have the metadata you need... so when you go back and try again to build your new installer today, it will already be there for you to use. Just download any random setuptools-based sdist from PyPI and run "tar -tzf" or "unzip -v" on it to see! For that matter, it's long enough ago that you should also have the fix already retroactively included in Distribute as well, so you should probably even be able to look at one of Distribute's own source distributions and see that it now contains the metadata you require. ;-) From regebro at gmail.com Sat May 29 19:18:05 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 29 May 2010 19:18:05 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> Message-ID: I'm not sure I understand this thread, mainly because I though distutils2 was to provide an installer. :) So with the risk of me not understanding it, here goes: I think both proposals, both the older bootstrap script and the summit proposal, is to make it easy to install an installer, which in practice means easy_install or pip. This is only necessary if we don't have a full installer in stdlib. I can see a couple of options: 0. We have a pyinstall script that runs pip *or* easy_install, depending on which one you choose when you first run it. I believe this is a bad idea, and will just create great amounts of confusion. 1. We include easy_install or pip in stdlib. However, I think we shouldn't include any installer in stdlib, until it has evolved into a proper package handling utility which also can uninstall, etc. 2. We include a pyinstall script that downloads from PyPI and runs setup.py install. This will cover 90% of install needs, for more advanced installers, you need to pyinstall pip or similar. This means that we end up with one more way to install, which isn't a disaster, but isn't really helping the situation. Also, the objection to including installers but not uninstallers are still relevant even though this is a super simplistic script that could be argues isn't really an installer at all. 3. We include a pypiget script, that will download a file from PyPI but *not* install it. That wouldn't create a one step install process, but it would speed up and simplify the install process without sacrificing flexibility. It also isn't an installer, so nobody can expect of require an uninstaller. But on the other hand, it's not exactly a massive helper. :-) 4. Write a proper installer/uninstaller and include that in stdlib. I suggest that it's call "pypi" and have commands like download/install/uninstall/upgrade. I'm not sure how much it needs to be based on/integrated with distutils2, as I don't know what the plans are for uninstaller support. Other than that it would pretty much only need to know how to download and run setup.py on packages, so I'm not sure it needs to be powered by distutils2 in any deeper sense. :-) Sorry if I'm answering something that wasn't a question. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ziade.tarek at gmail.com Sat May 29 21:12:24 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 29 May 2010 21:12:24 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> Message-ID: On Sat, May 29, 2010 at 6:32 PM, P.J. Eby wrote: [..] > > So? ?It's not like they're going to accidentally run 'easy_install' when > they meant to type 'pip', or vice versa. ?;-) They will accidentally run easy_install through a simple "python setup.py install" call if the given project bootstraps ez_setup and uses setuptools' easy_install command. The end-user is not asked in the process you describe, what installer he wants to use. It's implicitly done. That's the point I am trying to express: it's "implicit, embed, installers in project's setup.py" vs "an installer globally installed, knowing how to install projects that follows a given standard" Some project with ez_setup.py embed will get installed using setuptools own standard, and will also install setuptools of course in the end-user Python system. PEP 345 adds what setuptools had in its own standard on the top of distutils metadata, and one of the goal now is to be able to rely on this new standard without having to embed your own installer and hook it in your setup.py. I mean, why would we have different metadata standards ? with should we have different installers that are not interoperable ? why should people need to add some code in their setup.py to make sure their stuff is correctly installed ? > >> It also implies that the >> developer provides a smart setup.py script that embeds that bootstrap, >> and runs some code. > > Only if they need something beyond what the distutils provide. So what will happen when people uses the new metadata 1.2 ? (PEP 345) I know I don't want to ask the developer to add a special bootstrap in their project just because they used PEP 345 fields. That's the PEP 345-compatible installers job. > >> It will also allow distributions to be "dumb" envelopes with static >> metadata that are the same all the time, no matter which tool created >> them, and eventually remove setup.py in favor of statically described >> metadata using PEP 345. > > For packages with complex build requirements or distutils extensions (e.g. > numpy), this is unlikely to happen any time soon. > > Conversely, for packages where this *is* the case, the current distutils is > adequate, and having a bootstrapper that can install them would be a win. I don't understand where the win is in this case. For me there are two distinct things: - python projects, following some standards and conventions - installers, that know how to install projects that follow some standards and conventions > > >> Today, for instance, if an installer wants to install a distribution >> based on setuptools, it has to run the "egg_info" command to extract >> the metadata, on the target system. > > It's a good thing Guido loaned me the time machine so I could go back to > 2005 and fix that one, then: > > ?http://svn.python.org/view/sandbox/trunk/setuptools/setuptools/command/sdist.py?view=diff&r1=41093&r2=41094 > > Five years should be long enough ago that by now all the setuptools-based > source distributions on PyPI today will already have the metadata you > need... ?so when you go back and try again to build your new installer > today, it will already be there for you to use. ?Just download any random > setuptools-based sdist from PyPI and run "tar -tzf" or "unzip -v" on it to > see! This egg-info is useless AFAIK. The egg_info command *has* to be run again because it can change depending on the running platform and the code in setup.py. The one in the source distribution is not guaranteed to be accurate. Moreover, you can't get them at PyPI unless you download the tarball. PEP 345 is different in that aspect because environment-specific metadata fields can be described. And, you can query those metadata at PyPI. As a matter of fact it's already the case: PyPI is now Metadata 1.2 compatible. You can do XML-RPC queries at PyPI to get the metadata 1.2 fields (so the dependencies) for the Distutils2 project. Regards Tarek -- Tarek Ziad? | http://ziade.org From merwok at netwok.org Sun May 30 01:18:09 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 30 May 2010 01:18:09 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> Message-ID: <4C01A0B1.8090706@netwok.org> Hi [Tarek] >> I was not thinking about this proposal. If this what Guido proposed at >> the summit, then I misunderstood. [PJE] > I don't know what he proposed at the summit - I'm referring to the > bootstrap script he wrote that actually does this. It was a couple > years ago on Python-Dev, IIRC. Could be this: http://mail.python.org/pipermail/python-dev/2008-March/077837.html Cheers From carl at oddbird.net Sun May 30 19:13:48 2010 From: carl at oddbird.net (Carl Meyer) Date: Sun, 30 May 2010 13:13:48 -0400 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> Message-ID: <4C029CCC.4080508@oddbird.net> Lennart Regebro wrote: > 1. We include easy_install or pip in stdlib. However, I think we > shouldn't include any installer in stdlib, until it has evolved into a > proper package handling utility which also can uninstall, etc. You are aware that pip uninstalls? It also queries PyPI, lists locally installed packages, etc. It at least has a start on most of the things one would want a "proper package handling utility" to do. > 4. Write a proper installer/uninstaller and include that in stdlib. I > suggest that it's call "pypi" and have commands like > download/install/uninstall/upgrade. I'm not sure how much it needs to > be based on/integrated with distutils2, as I don't know what the plans > are for uninstaller support. Other than that it would pretty much only > need to know how to download and run setup.py on packages, so I'm not > sure it needs to be powered by distutils2 in any deeper sense. :-) This sounds like a pip clone. Pip's installation "interface" is nothing more than "setup.py install" in a subshell, so distutils2 support for installation will likely be quite simple. Replacing pip's use of pkg_resources with distutils2's pkgutil for querying local package info will be a bit more work, but the APIs are pretty similar. The question about a hypothetical "distutils2 installer" is backwards compatibility. If it is not backwards-compatible with existing distutils/setuptools packages, it will be useless to most people (there will be legacy packages hanging around, ones that people depend on, for many years to come). If it is backwards-compatible with existing packages, then it is a reimplementation of pip. I sympathize with the ease-of-use reasons for having something in the stdlib. And I'm not sure putting pip in the stdlib would be good for pip's development. So I don't have a solution to propose. But I have a hard time believing that reimplementing pip is even under serious consideration. It smells of seriously underestimating the amount of work involved. Carl From ziade.tarek at gmail.com Sun May 30 22:57:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 30 May 2010 22:57:20 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <4C029CCC.4080508@oddbird.net> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> Message-ID: On Sun, May 30, 2010 at 7:13 PM, Carl Meyer wrote: [..] > > I sympathize with the ease-of-use reasons for having something in the > stdlib. And I'm not sure putting pip in the stdlib would be good for > pip's development. So I don't have a solution to propose. But I have a > hard time believing that reimplementing pip is even under serious > consideration. It smells of seriously underestimating the amount of work > involved. I won't speak for others, but on my side, I don't underestimate the amount of work involved in an installer (heck, if I was scared by the amount of work in packaging, I'd quit 2 years ago ;)), but rather trying to figure out what is the path for a better packaging ecosystem. (it's a bit of a wicked problem) If you put aside the backward compatibility issues for a moment, I see several problems right now: A - If distutils2 doesn't include an installer, a fresh Python install will not be fully functional. I don't think it's the best option. Python should provide an installer/uninstaller that is compatible with the latest standards distutils2 provides. I don't want to release a software that will have to rely on a third party installer which don't exists yet : there's no clear plan to support the new standards yet in pip or in any other project afaik. B - adding within distutils2 an installer like pip (with the support of new standards) at a given version doesn't mean you can't develop the next version and release it as a third party package, so people can upgrade their installer and get the bleeding edge thing. This is doable as long as the installer is configurable. C - easy_install for instance, is already installed in Mac OS X by default. That makes sense from a end-user pov. So why wouldn't we have an installer in distutils2 in the stdlib ? D - letting *each* distribution decide what installer will be used when "python setup.py install" is called, without letting the end-user decides, (that's the ez_setup embed thing) means that we bury our goal to get rid of this file and to move toward statically defined metadata+ content. IOW have a clean separation between a Distribution and the code that is used to install it (eg the installer). Now about the backward compatibility issues: - as discussed with Ian, we are going to add backward compatibility in pkgutil, so we can read old egg-info formats. IOW, it deprecates the usage of pkg_resources in favor of an unified API. - if the installer part of distutils2 includes backward compatibility, it's ok I guess, as long as it's isolated to that part. So, I am still proposing to merge both projects and to have two kinds of releases: - distutils2 + pip included (maybe a renamed script) - pip, alone. And with the new metadata we have "provides-dist", so we can avoid conflicts here. Regards, Tarek -- Tarek Ziad? | http://ziade.org From pje at telecommunity.com Sun May 30 23:17:49 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 30 May 2010 17:17:49 -0400 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> Message-ID: <20100530211753.03E863A402D@sparrow.telecommunity.com> At 09:12 PM 5/29/2010 +0200, Tarek Ziad? wrote: > > For packages with complex build requirements or distutils extensions (e.g. > > numpy), this is unlikely to happen any time soon. > > > > Conversely, for packages where this *is* the case, the current distutils is > > adequate, and having a bootstrapper that can install them would be a win. > >I don't understand where the win is in this case. That packages with complex build requirements will still work. See above. From regebro at gmail.com Mon May 31 09:51:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 31 May 2010 09:51:17 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <4C029CCC.4080508@oddbird.net> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> Message-ID: On Sun, May 30, 2010 at 19:13, Carl Meyer wrote: > Lennart Regebro wrote: >> 1. We include easy_install or pip in stdlib. However, I think we >> shouldn't include any installer in stdlib, until it has evolved into a >> proper package handling utility which also can uninstall, etc. > > You are aware that pip uninstalls? Not reliably, as it doesn't keep track of files, so it can't remove anything installed outside of the package. > It also queries PyPI, lists locally > installed packages, etc. It at least has a start on most of the things > one would want a "proper package handling utility" to do. Absolutely. > This sounds like a pip clone. Indeed. I should have clarified that the name "pypi" I suggested was to satisfy up Perl people claiming that Python don't have CPAN. ;-) I decided against saying that to not make anyone upset, seems I failed. I also did not suggest the inclusion of pip not not make easy_install fans angry. Seriously, all packaging people: You need to be less tetchy. Not every single debate needs to result in a flame war. :-) I'm serious. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Mon May 31 09:55:03 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 31 May 2010 09:55:03 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> Message-ID: On Sun, May 30, 2010 at 22:57, Tarek Ziad? wrote: > So, I am still proposing to merge both projects and to have two kinds > of releases: > > - distutils2 + pip included (maybe a renamed script) I don't think this is necessary. Sure, distutils without an installer is kinda crippled, but the people who want to install distutils2 are capable of installing pip as well. :-) And an install script, like the one for distribute, could download and install both anyway. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From carl at oddbird.net Mon May 31 16:43:42 2010 From: carl at oddbird.net (Carl Meyer) Date: Mon, 31 May 2010 10:43:42 -0400 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> Message-ID: <4C03CB1E.2030507@oddbird.net> Tarek Ziad? wrote: > I won't speak for others, but on my side, I don't underestimate the > amount of work involved in an installer (heck, if I was scared by the > amount of work in packaging, I'd quit 2 years ago ;)), but rather > trying to figure out what is the path for a better packaging > ecosystem. (it's a bit of a wicked problem) Indeed! (If anyone's put in enough time in packaging in the last couple years to make writing a new installer from scratch seem possible; it would be you. I hope we can find a solution that allows that energy to be put into something more productive, however!) > A - If distutils2 doesn't include an installer, a fresh Python install > will not be fully functional. I don't think it's the best option. > Python should provide an installer/uninstaller that is compatible with > the latest standards distutils2 provides. I don't want to release a > software that will have to rely on a third party installer which don't > exists yet : there's no clear plan to support the new standards yet in > pip or in any other project afaik. I agree that it makes sense for there to be an installer packaged with distutils2 that supports all the features of the metadata standard, including dependencies. As far as "clear plan," what sort of plan are you looking for? The direction of pip mainline is up to Ian, of course, but I personally plan to do what I can towards a branch of pip that works with the new standards by the time distutils2 is released with a Python release. Given that pip will largely sit "atop" distutils2, I don't think it makes sense to start on the work in pip until distutils2 has stabilized. > B - adding within distutils2 an installer like pip (with the support > of new standards) at a given version doesn't mean you can't develop > the next version and release it as a third party package, so people > can upgrade their installer and get the bleeding edge thing. > This is doable as long as the installer is configurable. Yep. > Now about the backward compatibility issues: > - as discussed with Ian, we are going to add backward compatibility in > pkgutil, so we > can read old egg-info formats. IOW, it deprecates the usage of > pkg_resources in favor of an unified API. Great! I must have missed the end of that discussion in IRC. > - if the installer part of distutils2 includes backward compatibility, > it's ok I guess, as long as it's isolated to that part. If the pkgutil API is integrated and backwards-compatible, the remaining backwards-compatibility considerations would be: 1) Package installation. This shouldn't be too difficult ("setup.py install"), though if we want to be able to install old packages using the new PEP 376 installation format, that would involve some work. 2) Package-finding. I don't know if there's been discussion yet about whether the new distutils2 world will continue the same liberal link-searching algorithms that easy_install pioneered, or whether there will be a move towards something more structured. > So, I am still proposing to merge both projects and to have two kinds > of releases: > > - distutils2 + pip included (maybe a renamed script) > - pip, alone. This sounds good to me. I don't really care what the included one is called, though to me renaming sounds mostly like asking for confusion. Carl From carl at oddbird.net Mon May 31 16:55:09 2010 From: carl at oddbird.net (Carl Meyer) Date: Mon, 31 May 2010 10:55:09 -0400 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> Message-ID: <4C03CDCD.1060300@oddbird.net> Hi Lennart, Lennart Regebro wrote: >> You are aware that pip uninstalls? > > Not reliably, as it doesn't keep track of files, so it can't remove > anything installed outside of the package. Not true, inasmuch as it depends on pip. When pip installs distributions it uses the --record option to keep a list of all installed files. When pip uninstalls distributions it installed, it reliably uninstalls everything (though it doesn't have the PEP 376 hash cleverness available yet to detect modified files). When pip is asked to uninstall packages installed via other methods that do not provide the full installed-files record, it does so on a best-effort basis. In the case of easy_installed packages, that means it uninstalls installed packages and console scripts; in most cases, this is everything that was installed. > Indeed. I should have clarified that the name "pypi" I suggested was > to satisfy up Perl people claiming that Python don't have CPAN. ;-) I > decided against saying that to not make anyone upset, seems I failed. > I also did not suggest the inclusion of pip not not make easy_install > fans angry. > > Seriously, all packaging people: You need to be less tetchy. Not every > single debate needs to result in a flame war. :-) I'm serious. Sorry! I wouldn't say I was upset, just concerned that unnecessary duplication of work be avoided if possible. I am not concerned in some territorial way about the name of the installer that's included; "pypi" is just fine with me. I am concerned that it not be needlessly reimplemented from scratch; we have enough work to do without that! Carl From ianb at colorstudy.com Mon May 31 16:55:20 2010 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 31 May 2010 09:55:20 -0500 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <4C029CCC.4080508@oddbird.net> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> Message-ID: On Sun, May 30, 2010 at 12:13 PM, Carl Meyer wrote: > I sympathize with the ease-of-use reasons for having something in the > stdlib. And I'm not sure putting pip in the stdlib would be good for > pip's development. > This is my primary concern. I have no problem with merging pip with distutils2, so that pip can share utility code, etc. While it would still have to be backward compatible with old packages, I don't think that would have to change just because it merged with distutils2. But... the standard library is where I get stuck. And release cycles. pip has its own release cycle right now. It's not particularly formal, but it's a fairly regular release schedule. In terms of scope I don't think pip is finished. For instance, at least pip should have querying abilities (similar to what yolk does). Also the standard library's backward compatibility guarantee's don't seem as relevant to an installer -- an installer is something you run at a certain point in time, it's not part of the runtime of a system itself, and so changes are not as much of a problem. At the same time it has to be compatible with packages written a long time ago... but it's a different sort of compatibility. Well, I could list the problems, but basically I strongly dislike the idea that, say, Python 3.2 will ship distutils2 with a certain version of pip, and that people would have to upgrade Python or wait for a Python release to get upgrades of pip. Or, if that's not the case, then we need to figure what it means to have distutils2 "in" the standard library while still making regular releases. If Python shipped a "install distutils2" script then that I could work with, because it would imply distutils2 was blessed (and whatever the version at the time of release was, it could possibly be installed) but without attaching release cycles. -- Ian Bicking | http://blog.ianbicking.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Mon May 31 17:11:42 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 31 May 2010 17:11:42 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> Message-ID: On Mon, May 31, 2010 at 4:55 PM, Ian Bicking wrote: [..] > Well, I could list the problems, but basically I strongly dislike the idea > that, say, Python 3.2 will ship distutils2 with a certain version of pip, > and that people would have to upgrade Python or wait for a Python release to > get upgrades of pip. .. or allow people to configure in distutils2 an alternative installer. That is: a newer version of Pip. So they don't suffer from the Python release cycle by being able to upgrade the installer. I don't think being able to upgrade the installer from within the stdlib breaks the stdlib stability promise, since an installer is not something people use in their code as a library. It doesn't really suffer from API deprecation: it's a high-level end-user tool that people use to install stuff. It's interface (that is, the command line options I guess) can be quite stable. If people are using some packaging API in their code, I'd expect them to be in the Distutils2 'toolbox' part, which can benefit from the stdlib stability. Then Pip-the-installer could rely on it, and maybe improve those API by having its own version for a while until the next Python version is released. Python would ship with a version of the installer that can be upgraded by people that know what they do. Tarek -- Tarek Ziad? | http://ziade.org From regebro at gmail.com Mon May 31 18:47:58 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 31 May 2010 18:47:58 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <4C03CDCD.1060300@oddbird.net> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> <4C03CDCD.1060300@oddbird.net> Message-ID: On Mon, May 31, 2010 at 16:55, Carl Meyer wrote: > Not true, inasmuch as it depends on pip. When pip installs distributions > it uses the --record option to keep a list of all installed files. When > pip uninstalls distributions it installed, it reliably uninstalls > everything (though it doesn't have the PEP 376 hash cleverness available > yet to detect modified files). Ah, OK, cool. That must be reasonably new, last time I saw a discussion about this it wasn't the case. That *could* be some time ago, though. :) From carl at oddbird.net Mon May 31 19:02:49 2010 From: carl at oddbird.net (Carl Meyer) Date: Mon, 31 May 2010 13:02:49 -0400 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> <4C03CDCD.1060300@oddbird.net> Message-ID: <4C03EBB9.10308@oddbird.net> Lennart Regebro wrote: > On Mon, May 31, 2010 at 16:55, Carl Meyer wrote: >> Not true, inasmuch as it depends on pip. When pip installs distributions >> it uses the --record option to keep a list of all installed files. When >> pip uninstalls distributions it installed, it reliably uninstalls >> everything (though it doesn't have the PEP 376 hash cleverness available >> yet to detect modified files). > > Ah, OK, cool. That must be reasonably new, last time I saw a > discussion about this it wasn't the case. That *could* be some time > ago, though. :) Nope, pip's used --record on installation for years, and the above has been true since the moment uninstall landed in pip. There are enough different ways things can get installed that it's not surprising that some discussions may have been confused ;-) Carl From regebro at gmail.com Mon May 31 19:04:30 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 31 May 2010 19:04:30 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <4C03EBB9.10308@oddbird.net> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> <4C03CDCD.1060300@oddbird.net> <4C03EBB9.10308@oddbird.net> Message-ID: On Mon, May 31, 2010 at 19:02, Carl Meyer wrote: > Nope, pip's used --record on installation for years, and the above has > been true since the moment uninstall landed in pip. There are enough > different ways things can get installed that it's not surprising that > some discussions may have been confused ;-) That may be it. Forcing --record in Python 3.2 would be a step forward then? :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ziade.tarek at gmail.com Mon May 31 19:09:04 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 31 May 2010 19:09:04 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <20100530211753.03E863A402D@sparrow.telecommunity.com> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <20100530211753.03E863A402D@sparrow.telecommunity.com> Message-ID: On Sun, May 30, 2010 at 11:17 PM, P.J. Eby wrote: > At 09:12 PM 5/29/2010 +0200, Tarek Ziad? wrote: >> >> > For packages with complex build requirements or distutils extensions >> > (e.g. >> > numpy), this is unlikely to happen any time soon. >> > >> > Conversely, for packages where this *is* the case, the current distutils >> > is >> > adequate, and having a bootstrapper that can install them would be a >> > win. >> >> I don't understand where the win is in this case. > > That packages with complex build requirements will still work. ?See above. Ok. Well yes, it'll be years before distutils2 is used in most projects. But, making new projects and converted projects avoid any code in setup.py when possible, thanks to PEP 345 and what we plan next to staticaly describe a package (see http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst) is the path I would like to follow. Now for C extensions and the likes, it's still very fuzzy because I don't think this part can always be handled through configuration. Some people have their own recipes to build extensions so.. We are working on the configure command for the GSOC (creates a configure file build/install can use), but automating/using a build environment on target systems is probably not a first citizen in Distutils. Maybe this part should be separated completely from setup.py, with a specific handling when a project is installed. Tarek -- Tarek Ziad? | http://ziade.org From carl at oddbird.net Mon May 31 19:10:43 2010 From: carl at oddbird.net (Carl Meyer) Date: Mon, 31 May 2010 13:10:43 -0400 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> <4C03CDCD.1060300@oddbird.net> <4C03EBB9.10308@oddbird.net> Message-ID: <4C03ED93.9090801@oddbird.net> Lennart Regebro wrote: > On Mon, May 31, 2010 at 19:02, Carl Meyer wrote: >> Nope, pip's used --record on installation for years, and the above has >> been true since the moment uninstall landed in pip. There are enough >> different ways things can get installed that it's not surprising that >> some discussions may have been confused ;-) > > That may be it. Forcing --record in Python 3.2 would be a step forward then? :-) Would be, except that it's a setuptools-only option. Pip is only able to use it for all packages because it forces the import of setuptools, and thus setuptools' command overrides. Fortunately we have the right answer in PEP 376, now we just need to get it implemented. Carl From ziade.tarek at gmail.com Mon May 31 19:10:53 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 31 May 2010 19:10:53 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> <4C03CDCD.1060300@oddbird.net> <4C03EBB9.10308@oddbird.net> Message-ID: On Mon, May 31, 2010 at 7:04 PM, Lennart Regebro wrote: > On Mon, May 31, 2010 at 19:02, Carl Meyer wrote: >> Nope, pip's used --record on installation for years, and the above has >> been true since the moment uninstall landed in pip. There are enough >> different ways things can get installed that it's not surprising that >> some discussions may have been confused ;-) > > That may be it. Forcing --record in Python 3.2 would be a step forward then? :-) You mean in the current distutils ? Because distutils2 will have the PEP 376 implementation, where we create a RECORD file for each installed project in its dist-info/ > > -- > Lennart Regebro: Python, Zope, Plone, Grok > http://regebro.wordpress.com/ > +33 661 58 14 64 > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From regebro at gmail.com Mon May 31 19:14:49 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 31 May 2010 19:14:49 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> <4C03CDCD.1060300@oddbird.net> <4C03EBB9.10308@oddbird.net> Message-ID: On Mon, May 31, 2010 at 19:10, Tarek Ziad? wrote: > You mean in the current distutils ? Because distutils2 will have the > PEP 376 implementation, > where we create a RECORD file for each installed project in its dist-info/ Good. I suffer from PEP overload, and have long since given up on remembering what has been decided. :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From ziade.tarek at gmail.com Mon May 31 20:49:57 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 31 May 2010 20:49:57 +0200 Subject: [Distutils] pip vs easy_install vs distutils2 In-Reply-To: <4C03ED93.9090801@oddbird.net> References: <20100528235508.DDBC93A411E@sparrow.telecommunity.com> <20100529163220.6A3BF3A405F@sparrow.telecommunity.com> <4C029CCC.4080508@oddbird.net> <4C03CDCD.1060300@oddbird.net> <4C03EBB9.10308@oddbird.net> <4C03ED93.9090801@oddbird.net> Message-ID: On Mon, May 31, 2010 at 7:10 PM, Carl Meyer wrote: > Lennart Regebro wrote: >> On Mon, May 31, 2010 at 19:02, Carl Meyer wrote: >>> Nope, pip's used --record on installation for years, and the above has >>> been true since the moment uninstall landed in pip. There are enough >>> different ways things can get installed that it's not surprising that >>> some discussions may have been confused ;-) >> >> That may be it. Forcing --record in Python 3.2 would be a step forward then? :-) > > Would be, except that it's a setuptools-only option. No, that's a distutils option. But setuptools has its own implementation, but they seem like clone as far as I can tell (I didn't digg into that yet) Tarek