From ben+python at benfinney.id.au Tue Dec 1 00:40:31 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 01 Dec 2009 10:40:31 +1100 Subject: [Distutils] PEP 345 - 3 new fields References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> Message-ID: <878wdnponk.fsf@benfinney.id.au> Tarek Ziad? writes: > So at the end, maybe we could just keep "Repository-Browse-URL" and > drop the other one unless > we can come up with some use cases for automatic processing of the url. That would be a good start, yes. It keeps clear the distinction between the repository URL versus the browse URL. -- \ ?Not to perambulate the corridors in the hours of repose in the | `\ boots of ascension.? ?ski hotel, Austria | _o__) | Ben Finney From floris.bruynooghe at gmail.com Tue Dec 1 01:05:16 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Tue, 1 Dec 2009 00:05:16 +0000 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> Message-ID: <20091201000516.GA18694@laurie.devork> On Mon, Nov 30, 2009 at 01:17:18PM -0600, Ian Bicking wrote: > On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziad? wrote: > > > As suggested in Catalog-SIG by some people, I would like to propose > > the addition of three more fields for 1.2: > > > > "Repository-URL" > > A string containing the URL for the package's repository. > > Example: > > http://svn.python.org/projects/python/trunk/ > > > > I like this in concept, but it's a bit vague as just a URL. What kind of > repository is it? Sniffing is possible, but doesn't work that well. For > pip I require something like hg+http://... (with exceptions for schemes that > are self-describing, like svn: and git:). Very good point, the reason I proposed these was because I know Debian uses them but I failed to properly look at their experience. Actually turns out that currently they use: Vcs-Svn, Vcs-Git, etc. as well as Vcs-Browser. But searching around they are also having this discussion and looking that changing that because it doens't scale (needing a new field for each new VCS...). How about two fields? Repository-VCS and Repository-URL where the first is something like "git", "hg", "svn" etc. > It's also unclear whether it should point to the trunk, a branch, or a tag > (related to the release). As a URL that's only even possible for svn, as > git/hg/bzr all point to the repository and URLs can't point to revisions, > tags, etc. (pip has some extensions to the URL format to accommodate these, > but they aren't formally defined at this time). It's probably fine to leave that up to the developer and not specify it, PyPI's just needs to display it to the user. Whatever the developer sees fit. Coming up with a schema that is and future proof and works with all VCSes out there will be pretty hard. Regards Floris From ben+python at benfinney.id.au Tue Dec 1 01:35:44 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 01 Dec 2009 11:35:44 +1100 Subject: [Distutils] PEP 345 - 3 new fields References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <20091201000516.GA18694@laurie.devork> Message-ID: <87ljhno7j3.fsf@benfinney.id.au> Floris Bruynooghe writes: > How about two fields? Repository-VCS and Repository-URL where the > first is something like "git", "hg", "svn" etc. Which still leaves the ?Repository-Browse-URL? field, making three in total. -- \ ?Science doesn't work by vote and it doesn't work by | `\ authority.? ?Richard Dawkins, _Big Mistake_ (The Guardian, | _o__) 2006-12-27) | Ben Finney From david.lyon at preisshare.net Tue Dec 1 04:12:49 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 30 Nov 2009 22:12:49 -0500 Subject: [Distutils] =?utf-8?q?Bundling_pkg=5Fresources?= In-Reply-To: <94bdd2610911301139g5e85bbbfy462073a209f362a7@mail.gmail.com> References: <94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com> <94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com> <20091130180608.4F0183A408B@sparrow.telecommunity.com> <94bdd2610911301139g5e85bbbfy462073a209f362a7@mail.gmail.com> Message-ID: On Mon, 30 Nov 2009 20:39:32 +0100, Tarek Ziad? wrote: > The other big difference is that if you happen to have another problem > with it in the future, you can count on an actively > maintained, community-driven project. (IOW not locked by a single > maintainer that has been AWOL for most of > the time in the past 2 years) haha ... sounds like distutils for windows.. I stopped counting how many user requests for distutils related functionality go unanswered on this list. Give us a break and stop bagging PJE. So what if he has less time or motivation to do work on setuptools. "AWOL" as in 'absent without leave' is actually a criminal accusation for which the punishment for that offence is gaol time in most countries. I certainly do not believe PJE is a criminal or that he is working in the military or has deserted his (paid) post leading to a potential to loss of life (as would be in the military). You've won - you've forked his project, have your own team. Why the need for public humiliation on top of that? People just want to help.. David From ianb at colorstudy.com Tue Dec 1 04:19:00 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 Nov 2009 21:19:00 -0600 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> Message-ID: On Mon, Nov 30, 2009 at 4:30 PM, Tarek Ziad? wrote: > On Mon, Nov 30, 2009 at 8:17 PM, Ian Bicking wrote: > > On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziad? > wrote: > >> > >> As suggested in Catalog-SIG by some people, I would like to propose > >> the addition of three more fields for 1.2: > >> > >> "Repository-URL" > >> A string containing the URL for the package's repository. > >> Example: > >> http://svn.python.org/projects/python/trunk/ > > > > I like this in concept, but it's a bit vague as just a URL. What kind of > > repository is it? Sniffing is possible, but doesn't work that well. For > > pip I require something like hg+http://... (with exceptions for schemes > that > > are self-describing, like svn: and git:). > > I didn't think that Pip would use this field to do some automatic process, > I think the use case was to simply display that url at PyPI. > > OTHO, Repository-Browse-URL already plays that role, so it would be > interesting to > provide in "Repository-URL", urls that could be processed by installers. > > Do you have a particular use case in mind ? I don't really. There is an active use case for links like: http://repo-location/MyPackage/trunk#egg=MyPackage-dev which are used to get the dev version. These links should go to a svn index (which both easy_install and pip understand specifically) or to some automatically-generated tarball (which I believe most of the other VCSs generate through their popular frontends, e.g., http://bitbucket.org/ianb/webob/get/tip.zip). If something more like that was added (more like Trunk-URL), that would be genuinely useful. Barring that, I'd recommend removing the Repository-URL field from the recommendation until we have a clearer use case. Right now, the better use case would be a field giving the checkout command, e.g.,: Repository-Checkout: hg clone http://bitbucket.org/ianb/webob But I don't think that is particularly useful, only that it is more useful than Repository-URL. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Tue Dec 1 16:11:32 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 1 Dec 2009 16:11:32 +0100 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> Message-ID: <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> On Tue, Dec 1, 2009 at 4:19 AM, Ian Bicking wrote: [..] > If something more like that was added (more like Trunk-URL), that would be > genuinely useful. ?Barring that, I'd recommend removing the Repository-URL > field from the recommendation until we have a clearer use case. ?Right now, > the better use case would be a field giving the checkout command, e.g.,: > ??Repository-Checkout: hg clone http://bitbucket.org/ianb/webob > But I don't think that is particularly useful, only that it is more useful > than Repository-URL. Here's a proposal then, that seems to synthetize what people have been saying: Let's drop "Repository-Browse-URL" and keep a single "Repository-URL" field, which is a free URL that can take any URL form. e.g. a browsable url, or a git/hg url etc. Now for "Change-Log" vs "Change-Log-URL", I think the first one is better, because that's what people are already doing in their packages (when they add their changelog at the end of their long_description option), and it's not hard for PyPI to store it and display it, besides Description. The value I see in that field is that: - it helps developers structurize a bit their project description. e.g. letting them separate the Description of the package from the changelog if any. - it will let PyPI separate the front page of the project from its changelog, e.g. in this page: http://pypi.python.org/pypi/zc.buildout, instead of having the changelog here, it could be a link to another page for the project. How does that sounds ? Tarek From benji at zope.com Tue Dec 1 16:27:34 2009 From: benji at zope.com (Benji York) Date: Tue, 1 Dec 2009 10:27:34 -0500 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> Message-ID: On Tue, Dec 1, 2009 at 10:11 AM, Tarek Ziad? wrote: > The value I see in that field is that: > > - it helps developers structurize a bit their project description. > e.g. letting them separate > ?the Description of the package from the changelog if any. Package creators can already do this by including a link to the change log if they so choose. > - it will let PyPI separate the front page of the project from its > changelog, e.g. in this page: > ?http://pypi.python.org/pypi/zc.buildout, instead of having the > changelog here, it could be a link > ?to another page for the project. Again, package creators can already do this. > How does that sounds ? I don't understand the value of this proposal. We don't need more overhead for package authors to wade through, even if they end up ignoring it. No... *especially* if they end up ignoring it. -- Benji York From ziade.tarek at gmail.com Tue Dec 1 16:50:52 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 1 Dec 2009 16:50:52 +0100 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> Message-ID: <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> On Tue, Dec 1, 2009 at 4:27 PM, Benji York wrote: > On Tue, Dec 1, 2009 at 10:11 AM, Tarek Ziad? wrote: >> The value I see in that field is that: >> >> - it helps developers structurize a bit their project description. >> e.g. letting them separate >> ?the Description of the package from the changelog if any. > > Package creators can already do this by including a link to the change > log if they so choose. > >> - it will let PyPI separate the front page of the project from its >> changelog, e.g. in this page: >> ?http://pypi.python.org/pypi/zc.buildout, instead of having the >> changelog here, it could be a link >> ?to another page for the project. > > Again, package creators can already do this. > >> How does that sounds ? > > I don't understand the value of this proposal. Structurizing the information. e.g. having a standardification of the information about the project, that is more detailed than a free text like we have in Description, and letting PyPI presenting this information in separated parts, rather than in one single block. Last, providing to the developers conventions to follow on the information they can provide about their projects, and how it'll be displayed at PyPI. IOW, expecting to find in the same place on a project's PyPI page, for every project uploaded at PyPI, a link to the repository, changelog and bug tracker. So I guess you are also against the addition of "Repository-URL" and "Bug-Tracker-URL" too ? Since those can be links in the long_description as well. Regards, Tarek From benji at zope.com Tue Dec 1 17:24:26 2009 From: benji at zope.com (Benji York) Date: Tue, 1 Dec 2009 11:24:26 -0500 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> Message-ID: On Tue, Dec 1, 2009 at 10:50 AM, Tarek Ziad? wrote: [snip a reiteration of the perceived benefits] > So I guess you are also against the addition of "Repository-URL" and > "Bug-Tracker-URL" too ? > Since those can be links in the long_description as well. I am. Don't get me wrong, I don't think these proposals will kill kittens or anything, it's just that every addition has a cost. I (apparently) see that cost as being higher than others do, therefore I feel the small benefit is swamped by the cost. -- Benji York From ziade.tarek at gmail.com Tue Dec 1 18:04:04 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 1 Dec 2009 18:04:04 +0100 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> Message-ID: <94bdd2610912010904l60bbcfe9jdf3a707c23045f02@mail.gmail.com> On Tue, Dec 1, 2009 at 5:24 PM, Benji York wrote: > On Tue, Dec 1, 2009 at 10:50 AM, Tarek Ziad? wrote: > [snip a reiteration of the perceived benefits] > >> So I guess you are also against the addition of "Repository-URL" and >> "Bug-Tracker-URL" too ? >> Since those can be links in the long_description as well. > > I am. > > Don't get me wrong, I don't think these proposals will kill kittens or > anything, it's just that every addition has a cost. ?I (apparently) see > that cost as being higher than others do, therefore I feel the small > benefit is swamped by the cost. What is the cost you are seeing in having new additional fields ? Existing projects can leave everything in their long_description if they want, nothing is lost. From ianb at colorstudy.com Tue Dec 1 18:21:14 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 1 Dec 2009 11:21:14 -0600 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> Message-ID: On Tue, Dec 1, 2009 at 9:11 AM, Tarek Ziad? wrote: > On Tue, Dec 1, 2009 at 4:19 AM, Ian Bicking wrote: > [..] > > If something more like that was added (more like Trunk-URL), that would > be > > genuinely useful. Barring that, I'd recommend removing the > Repository-URL > > field from the recommendation until we have a clearer use case. Right > now, > > the better use case would be a field giving the checkout command, e.g.,: > > Repository-Checkout: hg clone http://bitbucket.org/ianb/webob > > But I don't think that is particularly useful, only that it is more > useful > > than Repository-URL. > > Here's a proposal then, that seems to synthetize what people have been > saying: > > Let's drop "Repository-Browse-URL" and keep a single "Repository-URL" > field, which is a free > URL that can take any URL form. e.g. a browsable url, or a git/hg url etc. > I prefer Repository-Browse-URL, as it is more explicit in its use: it's a link that someone using a browser can usefully click through to. I expect it will be displayed as such on PyPI. So this link is good: http://github.com/cloudkick/libcloud And this link is bad: git://github.com/cloudkick/libcloud.git A similar distinction exists for code.google.com projects. Now for "Change-Log" vs "Change-Log-URL", I think the first one is > better, because > that's what people are already doing in their packages (when they add > their changelog at the > end of their long_description option), and it's not hard for PyPI to > store it and display it, besides > Description. > This seems fine to me. Is ReST allowed? Could one potentially just do: `Changes `_ ? And then essentially the changelog would be a single link? I'm not sure if that's a good idea. Would it be too vague to say that if the change log is a single URL that PyPI should link directly through to the change log instead of displaying the link? (The exact UI for PyPI hasn't been proposed, but if it's something like a tab with changes, that tab could actually link offsite) -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker -------------- next part -------------- An HTML attachment was scrubbed... URL: From benji at zope.com Tue Dec 1 19:27:09 2009 From: benji at zope.com (Benji York) Date: Tue, 1 Dec 2009 13:27:09 -0500 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <94bdd2610912010904l60bbcfe9jdf3a707c23045f02@mail.gmail.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> <94bdd2610912010904l60bbcfe9jdf3a707c23045f02@mail.gmail.com> Message-ID: On Tue, Dec 1, 2009 at 12:04 PM, Tarek Ziad? wrote: > What is the cost you are seeing in having new additional fields ? The cognitive load of package authors having to read, understand, and then provide or ignore the metadata. There is also the cost to implementors of PyPI replacements/mirrors/other tools. > Existing projects can leave everything in their long_description if > they want, nothing is lost. Nothing we currently have is lost, however we put a small damper on future gains. -- Benji York From eric at trueblade.com Tue Dec 1 20:22:13 2009 From: eric at trueblade.com (Eric Smith) Date: Tue, 01 Dec 2009 14:22:13 -0500 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> Message-ID: <4B156CE5.4080200@trueblade.com> Benji York wrote: > On Tue, Dec 1, 2009 at 10:50 AM, Tarek Ziad? wrote: > [snip a reiteration of the perceived benefits] > >> So I guess you are also against the addition of "Repository-URL" and >> "Bug-Tracker-URL" too ? >> Since those can be links in the long_description as well. > > I am. > > Don't get me wrong, I don't think these proposals will kill kittens or > anything, it's just that every addition has a cost. I (apparently) see > that cost as being higher than others do, therefore I feel the small > benefit is swamped by the cost. I agree with Benji that adding new fields has a cost and should be avoided. Just because we can add structure doesn't mean we should. In the specific case of Repository-URL and Bug-Tracker-URL, the idea was that PyPI would change its behavior if these fields were present (by not using the rating system). So I'm okay with those. But for other fields, like Repository-Checkout and Change-Log-URL, I haven't seen a use case yet. Eric. From chris at simplistix.co.uk Wed Dec 2 10:37:46 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 02 Dec 2009 09:37:46 +0000 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <4B13C66D.4060206@trueblade.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <4B11A54D.1020706@simplistix.co.uk> <94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com> <4B138CBB.3000402@simplistix.co.uk> <4B13C66D.4060206@trueblade.com> Message-ID: <4B16356A.3090401@simplistix.co.uk> Eric Smith wrote: >> Having the option to put a url in a standard field would encourage >> people to just put a link to the changelog. > > Why does it need to be in a standard field? The same reason we have standard metadata fields like homepage, etc: so users of PyPI get a consistent location in which to find the information they're looking for, no matter who the package author is or how they choose to layout their long_description. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Wed Dec 2 10:49:08 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 02 Dec 2009 09:49:08 +0000 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <4B11A54D.1020706@simplistix.co.uk> <94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com> <871vjitbyl.fsf@benfinney.id.au> <94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com> Message-ID: <4B163814.2020700@simplistix.co.uk> Tarek Ziad? wrote: > How about making it "Change-Log" in that case, and have the content of > the changelog > directly added in the metadata ? Rather than an url. Well, I wouldn't object to that if it was added in addition to Change-Log-URL, but bear in mind this makes much more work for PyPI than Change-Log-URL. Change-Log requires a new page, and integration with the package page. Change-Log-URL just requires a new url field added on the existing package page. Change-Log-URL also gives the freedom for package maintainers to keep their changelog wherever they like and refer to it. This might not be on PyPI; my changelogs are becoming sphinx pages, and while those pages are currently on packages.python.org, they will eventually be on http://simplistix.couk/software. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Wed Dec 2 15:16:13 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 2 Dec 2009 15:16:13 +0100 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> Message-ID: <94bdd2610912020616p42411ad3qd113165e77730b7c@mail.gmail.com> On Tue, Dec 1, 2009 at 6:21 PM, Ian Bicking wrote: [...] >> >> Here's a proposal then, that seems to synthetize what people have been >> saying: >> >> Let's drop "Repository-Browse-URL" and keep a single "Repository-URL" >> field, which is a free >> URL that can take any URL form. e.g. a browsable url, or a git/hg url etc. > > I prefer Repository-Browse-URL, as it is more explicit in its use: it's a > link that someone using a browser can usefully click through to. ?I expect > it will be displayed as such on PyPI. ?So this link is good: > ??http://github.com/cloudkick/libcloud > > And this link is bad: > ??git://github.com/cloudkick/libcloud.git > A similar distinction exists for code.google.com projects. Right, >> Now for "Change-Log" vs "Change-Log-URL", I think the first one is >> better, because >> that's what people are already doing in their packages (when they add >> their changelog at the >> end of their long_description option), and it's not hard for PyPI to >> store it and display it, besides >> Description. > > This seems fine to me. ?Is ReST allowed? ?Could one potentially just do: > ??`Changes `_ > ? ?And then essentially the changelog would be a single link? ?I'm not sure > if that's a good idea. ?Would it be too vague to say that if the change log > is a single URL that PyPI should link directly through to the change log > instead of displaying the link? ?(The exact UI for PyPI hasn't been > proposed, but if it's something like a tab with changes, that tab could > actually link offsite) The idea is to have a similar field than Description, i.e. a free text with reST allowed, so it can be potentially parsed by PyPI. Tarek From ziade.tarek at gmail.com Wed Dec 2 15:43:28 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 2 Dec 2009 15:43:28 +0100 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <4B156CE5.4080200@trueblade.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> <4B156CE5.4080200@trueblade.com> Message-ID: <94bdd2610912020643x4979021ete9313642176d2113@mail.gmail.com> 2009/12/1 Eric Smith : > Benji York wrote: >> >> On Tue, Dec 1, 2009 at 10:50 AM, Tarek Ziad? >> wrote: >> [snip a reiteration of the perceived benefits] >> >>> So I guess you are also against the addition of "Repository-URL" and >>> "Bug-Tracker-URL" too ? >>> Since those can be links in the long_description as well. >> >> I am. >> >> Don't get me wrong, I don't think these proposals will kill kittens or >> anything, it's just that every addition has a cost. ?I (apparently) see >> that cost as being higher than others do, therefore I feel the small >> benefit is swamped by the cost. > > I agree with Benji that adding new fields has a cost and should be avoided. > Just because we can add structure doesn't mean we should. > > In the specific case of Repository-URL and Bug-Tracker-URL, the idea was > that PyPI would change its behavior if these fields were present (by not > using the rating system). So I'm okay with those. But for other fields, like > Repository-Checkout and Change-Log-URL, I haven't seen a use case yet. Ok so, if everyone agrees on those two, I am proposing to add for now : - Repository-Browse-URL : url to a web-browseable repository (no distinction whether its the trunk, a branch etc) - Bug-Tracker-URL : url to the web tracker and skip the others, as they seem controversial. I'll post the proposal at Catalog-SIG as well, Tarek From floris.bruynooghe at gmail.com Thu Dec 3 00:01:09 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Wed, 2 Dec 2009 23:01:09 +0000 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <94bdd2610912020643x4979021ete9313642176d2113@mail.gmail.com> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> <4B156CE5.4080200@trueblade.com> <94bdd2610912020643x4979021ete9313642176d2113@mail.gmail.com> Message-ID: <20091202230109.GB15234@laurie.devork> On Wed, Dec 02, 2009 at 03:43:28PM +0100, Tarek Ziad? wrote: > Ok so, if everyone agrees on those two, I am proposing to add for now : > > - Repository-Browse-URL : url to a web-browseable repository (no > distinction whether its the trunk, > a branch etc) I'm still uneasy about forcing everyone to have a browsable URL and would rather just have a Repository-URL which is reccomended to be browsable but doens't have to. But if I'm in the minority then so be it. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From ben+python at benfinney.id.au Thu Dec 3 00:08:28 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 03 Dec 2009 10:08:28 +1100 Subject: [Distutils] PEP 345 - 3 new fields References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> <4B156CE5.4080200@trueblade.com> <94bdd2610912020643x4979021ete9313642176d2113@mail.gmail.com> <20091202230109.GB15234@laurie.devork> Message-ID: <87pr6xkm8j.fsf@benfinney.id.au> Floris Bruynooghe writes: > I'm still uneasy about forcing everyone to have a browsable URL I think the intention is that all of these new fields are optional. No-one would be forced to put any of them into their package. -- \ ?Every sentence I utter must be understood not as an | `\ affirmation, but as a question.? ?Niels Bohr | _o__) | Ben Finney From dalke at dalkescientific.com Thu Dec 3 01:06:51 2009 From: dalke at dalkescientific.com (Andrew Dalke) Date: Thu, 3 Dec 2009 01:06:51 +0100 Subject: [Distutils] installing .py plugins to an alternate directory Message-ID: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> Hi all, I'm working with the Akara project. It contains a web server. The server loads extensions from a special directory (let's say "$AKARA" for now). An extension can register handlers for URLs. An example extension might look like: installs to $AKARA/spam_extension.py (note: only .py files are supported; not even .pyc files) ========================= from akara.services import simple_service import my_spam # This is part of the distribution, and gets put in site-packages @simple_service("GET", "http://vikings.protocol.id/") def vikings(say=my_spam.DEFAULT_TEXT): return my_spam.vikings(say) ========================= We want people to be able to distribute Akara plugins and install via setup.py. Ideally I would like to say: from distutils.core import setup from akara.distutils ... I'm not sure what here ... setup(name="Spam services", package="my_spam", akara_extensions=["spam_extension.py"] ) To clarify, the development/distribution package looks like: $PACKAGE/setup.py $PACKAGE/README $PACKAGE/spam_extensions.py $PACKAGE/my_spam/__init__.py $PACKAGE/my_spam/dramatis_personae.py $PACKAGE/my_spam/cafe.py and $PACKAGE/spam_extensions.py goes to $AKARA/spam_extensions.py while $PACKAGE/my_spam is copied to site-packages. The installation does not need to byte-compile spam_extension.py. It should also include spam_extension.py in any distribution that it makes. I looked through the documentation and searched for existing examples, but found nothing which does this. The plugins I found used entry_points, and that's an architecture change which I don't think is appropriate for us. Suggestions? I had hoped that I could add my own cmdclasses like "build_akara" and "install_akara" which would get called during the correct stages of the build process, but that seems to be a dead end. I might be able to hack the "install_data" cmdclass to make it work, but that would prevent anyone from using it in their own setup.py distributions. Andrew dalke at dalkescientific.com From david at ar.media.kyoto-u.ac.jp Thu Dec 3 04:46:53 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 03 Dec 2009 12:46:53 +0900 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> Message-ID: <4B1734AD.10500@ar.media.kyoto-u.ac.jp> Andrew Dalke wrote: > Hi all, > > I'm working with the Akara project. It contains a web server. The server loads extensions from a special directory (let's say "$AKARA" for now). An extension can register handlers for URLs. An example extension might look like: > > installs to $AKARA/spam_extension.py > (note: only .py files are supported; not even .pyc files) > ========================= > from akara.services import simple_service > > import my_spam # This is part of the distribution, and gets put in site-packages > > @simple_service("GET", "http://vikings.protocol.id/") > def vikings(say=my_spam.DEFAULT_TEXT): > return my_spam.vikings(say) > ========================= > > > We want people to be able to distribute Akara plugins and install via setup.py. Ideally I would like to say: > > from distutils.core import setup > from akara.distutils ... I'm not sure what here ... > > setup(name="Spam services", > package="my_spam", > akara_extensions=["spam_extension.py"] > ) > > > To clarify, the development/distribution package looks like: > $PACKAGE/setup.py > $PACKAGE/README > $PACKAGE/spam_extensions.py > $PACKAGE/my_spam/__init__.py > $PACKAGE/my_spam/dramatis_personae.py > $PACKAGE/my_spam/cafe.py > > and $PACKAGE/spam_extensions.py goes to $AKARA/spam_extensions.py while $PACKAGE/my_spam is copied to site-packages. > > The installation does not need to byte-compile spam_extension.py. > > It should also include spam_extension.py in any distribution that it makes. > > > I looked through the documentation and searched for existing examples, but found nothing which does this. The plugins I found used entry_points, and that's an architecture change which I don't think is appropriate for us. I won't comment on entry_points, as I have never used them. The way I would do it is by having akara distutils extensions, which define in particular a setup function and associated classes. Here is the code: import shutil from distutils.core import setup as _setup from distutils.core import Command from distutils import log # XXX: you will need to handle setuptools (and distribute, and....) # monkey-patching for the below commands from distutils.command.install import install as old_install from distutils.dist import Distribution as _Distribution # Where to install akara extensions AKARA_SITE = '/tmp' def setup(**kw): new_kw = kw.copy() # Handle commands overriding cmdclass = new_kw.get('cmdclass', {}) if 'install' in cmdclass: install = cmdclass['install'] else: install = old_install class my_install(install): sub_commands = install.sub_commands + [ ('install_akara_extensions', lambda x: True) ] class install_akara_extensions(Command): description = "Command to install akara extensions" user_options = [ ('akara-site=', None, 'Akara plugin directory install'), ] def initialize_options(self): self.install_dir = None self.akara_site = None self.outfiles = [] def finalize_options(self): if self.akara_site is None: self.akara_site = AKARA_SITE def run (self): dist = self.distribution for a in dist.akara_extensions: log.info("Installing akara extension %s in %s" % (a, self.akara_site)) shutil.copy(a, self.akara_site) new_cmdclass = {} new_cmdclass.update(cmdclass) new_cmdclass['install'] = my_install new_cmdclass['install_akara_extensions'] = install_akara_extensions new_kw['cmdclass'] = new_cmdclass # Handle overriden distclass if not 'distclass' in new_kw: Distribution = new_kw['distclass'] else: Distribution = _Distribution class MyDistribution(Distribution): def __init__(self, attrs=None): assert not hasattr(self, 'akara_extensions') if 'akara_extensions' in attrs: self.akara_extensions = attrs['akara_extensions'] else: self.akara_extensions = [] Distribution.__init__(self, attrs) new_kw['distclass'] = MyDistribution return _setup(**new_kw) setup(name="Spam services", packages=["my_spam"], akara_extensions=["spam_extension.py"] ) This is the minimal code to support this semi correctly. Note that many things are not handled correctly: - get_output for install_akara_extensions should be handled - most likely it will break for semi-random distutils extensions, including setuptools and distribute: you will need to detect monkey-patching to decide which Distribution and install commands to override. I have not done it because it requires the extensions to be in a separate package (to conditionally import things depending on detected monkey patching). If you think this is insane, you are not alone :) David From mal at egenix.com Thu Dec 3 13:55:53 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 03 Dec 2009 13:55:53 +0100 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com> <94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com> <4B0D991D.3020502@egenix.com> <94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com> <4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com> <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> Message-ID: <4B17B559.1030800@egenix.com> Tarek Ziad? wrote: > Last, as I said in a previous mail, I tend to agree with the people > who said that we should stick with only one way to write the version > scheme for the sake of clarity. e.g. dropping aliases and picking > *one* way to write the markers after major.minor.micro. > > I would tend to pick the same scheme than Python for the pre-releases > (and c + rc): > > N.N[.N][(a|b|c|rc)N] > > And, for the post/dev markers I think dots are OK for readability, Sure, but readability and clarity means different things for different people. The reason I proposed aliases and underscores is to give package authors the choice of using terse forms or more verbose ones, as well as making the whole scheme more compatible to existing version strings already in use. Regarding post/dev markers: IMO, it's not really obvious that a 1.0a1.dev123 release refers to a snaphost *before* the 1.0a1 release. The string "pre" is more commonly used for such pre-release snapshots. For the .post123 tag, I don't see a need for a "post" string at all, 1.0a1.123 is clearly a release or build *after* the 1.0a1 release and since the "1.123" is being treated as alpha version number, the post part processing can be dropped altogether. For the .dev part the situation is similar: you can always choose a pre-release version that is not actually released and then issue follow up snapshots to this, e.g. 1.0a0.20091203 1.0a0.20091204 1.0a0.20091205 and so on for nightly builds during the development phase. Instead of writing: 1.0a1.dev20091205 you'd then write 1.0a0.20091205 This is how Python itself is organizing the versions during development, BTW. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From a.badger at gmail.com Thu Dec 3 16:17:17 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 3 Dec 2009 07:17:17 -0800 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <4B17B559.1030800@egenix.com> References: <4B0D991D.3020502@egenix.com> <94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com> <4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com> <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> <4B17B559.1030800@egenix.com> Message-ID: <20091203151717.GA3471@clingman.lan> On Thu, Dec 03, 2009 at 01:55:53PM +0100, M.-A. Lemburg wrote: > Tarek Ziad? wrote: > > Last, as I said in a previous mail, I tend to agree with the people > > who said that we should stick with only one way to write the version > > scheme for the sake of clarity. e.g. dropping aliases and picking > > *one* way to write the markers after major.minor.micro. > > > > I would tend to pick the same scheme than Python for the pre-releases > > (and c + rc): > > > > N.N[.N][(a|b|c|rc)N] > > > > And, for the post/dev markers I think dots are OK for readability, > > Sure, but readability and clarity means different things for > different people. > > The reason I proposed aliases and underscores is to give package > authors the choice of using terse forms or more verbose ones, as > well as making the whole scheme more compatible to existing > version strings already in use. > I'm not a big fan of underscores -- having multiple separators doesn't seem very useful. I don'tlike aliases but seeing as I like the long forms, having both short and long but giving them a distinct ordering would be okay to me (ie: a1 < alpha1 < a2 < b1 < beta1 < c1 < rc1 > Regarding post/dev markers: > > IMO, it's not really obvious that a 1.0a1.dev123 release refers to a > snaphost *before* the 1.0a1 release. The string "pre" is more commonly > used for such pre-release snapshots. > > For the .post123 tag, I don't see a need for a "post" string at all, > 1.0a1.123 is clearly a release or build *after* the 1.0a1 release > and since the "1.123" is being treated as alpha version number, > the post part processing can be dropped altogether. > > For the .dev part the situation is similar: you can always > choose a pre-release version that is not actually released and then > issue follow up snapshots to this, e.g. > > 1.0a0.20091203 > 1.0a0.20091204 > 1.0a0.20091205 > > and so on for nightly builds during the development phase. > > Instead of writing: > > 1.0a1.dev20091205 > > you'd then write > > 1.0a0.20091205 > > This is how Python itself is organizing the versions during > development, BTW. > FWIW, I agree with all of this section. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From ziade.tarek at gmail.com Thu Dec 3 16:18:10 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 3 Dec 2009 16:18:10 +0100 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <4B17B559.1030800@egenix.com> References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com> <94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com> <4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com> <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> <4B17B559.1030800@egenix.com> Message-ID: <94bdd2610912030718j2b79414ah74758b97805d11d8@mail.gmail.com> On Thu, Dec 3, 2009 at 1:55 PM, M.-A. Lemburg wrote: > Tarek Ziad? wrote: >> Last, as I said in a previous mail, I tend to agree with the people >> who said that we should stick with only one way to write the version >> scheme for the sake of clarity. e.g. dropping aliases and picking >> *one* way to write the markers after major.minor.micro. >> >> I would tend to pick the same scheme than Python for the pre-releases >> (and c + rc): >> >> ? ? N.N[.N][(a|b|c|rc)N] >> >> And, for the post/dev markers I think dots are OK for readability, > > Sure, but readability and clarity means different things for > different people. > > The reason I proposed aliases and underscores is to give package > authors the choice of using terse forms or more verbose ones, as > well as making the whole scheme more compatible to existing > version strings already in use. > > Regarding post/dev markers: > > IMO, it's not really obvious that a 1.0a1.dev123 release refers to a > snaphost *before* the 1.0a1 release. The string "pre" is more commonly > used for such pre-release snapshots. > > For the .post123 tag, I don't see a need for a "post" string at all, > 1.0a1.123 is clearly a release or build *after* the 1.0a1 release > and since the "1.123" is being treated as alpha version number, > the post part processing can be dropped altogether. > > For the .dev part the situation is similar: you can always > choose a pre-release version that is not actually released and then > issue follow up snapshots to this, e.g. > > ? ? ? ?1.0a0.20091203 > ? ? ? ?1.0a0.20091204 > ? ? ? ?1.0a0.20091205 > > and so on for nightly builds during the development phase. > > Instead of writing: > > ? ? ? ?1.0a1.dev20091205 > > you'd then write > > ? ? ? ?1.0a0.20091205 > > This is how Python itself is organizing the versions during > development, BTW. So IOW, are you suggesting that a suffix marker is always a post release marker ? so we have : 1.0a0 < 1.0a0.124 < 1.0a0.245 < 1.0a1 < 1.0a1.346 < 1.0a2 < 1.0 < 1.0.567 <--- dev marker The problem in that case is that we would be unable to differenciate dev marker with micro markers. That's why we added these "dev" markers in the first place. So following your ordering: - 1.0a0 < 1.0a0.dev124 < 1.0a0.dev245 < 1.0a1 < 1.0a1.dev346 < 1.0a2 < 1.0 < 1.0.dev567 <--- dev marker Now about making the dev a post release tag, AFAIK people always use dev markers as pre-release tags: - 1.0a0 < 1.0a1.dev124 < 1.0a1.dev245 < 1.0a1 < 1.0a2.dev346 < 1.0a2 < 1.0.dev567 <--- dev marker < 1.0 Last, if we want to do a post release for 1.0 for example, you would also need a post- marker, as we added Regards Tarek From mal at egenix.com Thu Dec 3 16:50:50 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 03 Dec 2009 16:50:50 +0100 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <94bdd2610912030718j2b79414ah74758b97805d11d8@mail.gmail.com> References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com> <94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com> <4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com> <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> <4B17B559.1030800@egenix.com> <94bdd2610912030718j2b79414ah74758b97805d11d8@mail.gmail.com> Message-ID: <4B17DE5A.8030604@egenix.com> Tarek Ziad? wrote: > On Thu, Dec 3, 2009 at 1:55 PM, M.-A. Lemburg wrote: >> Tarek Ziad? wrote: >>> Last, as I said in a previous mail, I tend to agree with the people >>> who said that we should stick with only one way to write the version >>> scheme for the sake of clarity. e.g. dropping aliases and picking >>> *one* way to write the markers after major.minor.micro. >>> >>> I would tend to pick the same scheme than Python for the pre-releases >>> (and c + rc): >>> >>> N.N[.N][(a|b|c|rc)N] >>> >>> And, for the post/dev markers I think dots are OK for readability, >> >> Sure, but readability and clarity means different things for >> different people. >> >> The reason I proposed aliases and underscores is to give package >> authors the choice of using terse forms or more verbose ones, as >> well as making the whole scheme more compatible to existing >> version strings already in use. >> >> Regarding post/dev markers: >> >> IMO, it's not really obvious that a 1.0a1.dev123 release refers to a >> snaphost *before* the 1.0a1 release. The string "pre" is more commonly >> used for such pre-release snapshots. >> >> For the .post123 tag, I don't see a need for a "post" string at all, >> 1.0a1.123 is clearly a release or build *after* the 1.0a1 release >> and since the "1.123" is being treated as alpha version number, >> the post part processing can be dropped altogether. >> >> For the .dev part the situation is similar: you can always >> choose a pre-release version that is not actually released and then >> issue follow up snapshots to this, e.g. >> >> 1.0a0.20091203 >> 1.0a0.20091204 >> 1.0a0.20091205 >> >> and so on for nightly builds during the development phase. >> >> Instead of writing: >> >> 1.0a1.dev20091205 >> >> you'd then write >> >> 1.0a0.20091205 >> >> This is how Python itself is organizing the versions during >> development, BTW. > > So IOW, are you suggesting that a suffix marker is always a post > release marker ? In a way, yes. For pre-releases, it's actually a minor revision of the pre-release version, e.g. in 1.0a0.123 the "0.123" part is the pre-release version. For releases, it's a normal part of the version number, e.g. in 1.0.123 the ".123" marks a patch level release. > so we have : > > 1.0a0 > < 1.0a0.124 > < 1.0a0.245 > < 1.0a1 > < 1.0a1.346 > < 1.0a2 > < 1.0 > < 1.0.567 <--- dev marker That's not a dev-marker, it's actually part of the release version: the patch level release number. > The problem in that case is that we would be unable to differenciate > dev marker with micro markers. The dev markers introduce an extra level of confusion, which IMHO is not necessary. Let's take 1.0a0.dev123 as example, reading it from the left: 1.0 - ok, so this is part of a 1.0 release 1.0a0 - oops, no, it's not actually part of a release, it's part of a pre-release, so move back 1.0a0.dev123 - hmm, not even that, it's part of what's going to become a pre-release, so move even further back As developer you typically want to use the version number to tell the user a few things about your package: 1. whether it's release quality code 1.0.0 2. whether it's a development snapshot 1.0.0a0.20091202 3. whether it's working code, but still under development 1.0.0a1 4. whether it fixes some bug that was found after a release 1.0.1 > That's why we added these "dev" markers in the first place. So > following your ordering: > > - 1.0a0 > < 1.0a0.dev124 > < 1.0a0.dev245 > < 1.0a1 > < 1.0a1.dev346 > < 1.0a2 > < 1.0 > < 1.0.dev567 <--- dev marker > > Now about making the dev a post release tag, AFAIK people always use > dev markers as pre-release tags: > > - 1.0a0 > < 1.0a1.dev124 > < 1.0a1.dev245 > < 1.0a1 > < 1.0a2.dev346 > < 1.0a2 > < 1.0.dev567 <--- dev marker > < 1.0 > > Last, if we want to do a post release for 1.0 for example, you would > also need a post- marker, as we added Sorry, I probably wasn't clear enough. What I proposed looks like this: 1.0a0 (which is not released and only used to mark the start of 1.0 development, just like we do for Python) < 1.0a0.124 < 1.0a0.245 < 1.0a1 (pre-release) < 1.0a1.346 < 1.0a2 (pre-release) < 1.0 (release) < 1.0.567 (release) and the 1.0.567 is not a dev-marker (since the proposal is about removing these ;-), but instead a post-marker without the "post". In reality, such a post release would be a patch-level release. If you want to then work on release 1.1, you'd continue with: - 1.0.567 < 1.1a0 (which is not released and only used to mark the start of 1.1 development, just like we do for Python) < 1.1a0.124 < 1.1a0.245 < 1.1a1 (pre-release) < 1.1a1.346 < 1.1a2 (pre-release) < 1.1 (release) As you can see, you don't need any dev-markers and post-markers turn into pre-release minor version numbers... less noise, more clarity, well at least IMHO. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Dec 3 16:53:37 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 03 Dec 2009 16:53:37 +0100 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <20091203151717.GA3471@clingman.lan> References: <4B0D991D.3020502@egenix.com> <94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com> <4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com> <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> <4B17B559.1030800@egenix.com> <20091203151717.GA3471@clingman.lan> Message-ID: <4B17DF01.9080104@egenix.com> Toshio Kuratomi wrote: > On Thu, Dec 03, 2009 at 01:55:53PM +0100, M.-A. Lemburg wrote: >> Tarek Ziad? wrote: >>> Last, as I said in a previous mail, I tend to agree with the people >>> who said that we should stick with only one way to write the version >>> scheme for the sake of clarity. e.g. dropping aliases and picking >>> *one* way to write the markers after major.minor.micro. >>> >>> I would tend to pick the same scheme than Python for the pre-releases >>> (and c + rc): >>> >>> N.N[.N][(a|b|c|rc)N] >>> >>> And, for the post/dev markers I think dots are OK for readability, >> >> Sure, but readability and clarity means different things for >> different people. >> >> The reason I proposed aliases and underscores is to give package >> authors the choice of using terse forms or more verbose ones, as >> well as making the whole scheme more compatible to existing >> version strings already in use. >> > I'm not a big fan of underscores -- having multiple separators doesn't seem > very useful. I'm not tied to those underscores, just think they'd help in making the strings more readable. > I don'tlike aliases but seeing as I like the long forms, having both short > and long but giving them a distinct ordering would be okay to me (ie: > > a1 < alpha1 < a2 < b1 < beta1 < c1 < rc1 That would also work. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Thu Dec 3 17:05:21 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 3 Dec 2009 17:05:21 +0100 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <4B17DE5A.8030604@egenix.com> References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com> <4B0E6FC2.5020006@egenix.com> <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> <4B17B559.1030800@egenix.com> <94bdd2610912030718j2b79414ah74758b97805d11d8@mail.gmail.com> <4B17DE5A.8030604@egenix.com> Message-ID: <94bdd2610912030805i52ffa5ah8f3eb5ad120f27d9@mail.gmail.com> [..] > 1. whether it's release quality code > > ? ? ? ?1.0.0 > > 2. whether it's a development snapshot > > ? ? ? ?1.0.0a0.20091202 > > 3. whether it's working code, but still under development > > ? ? ? ?1.0.0a1 > > 4. whether it fixes some bug that was found after a release > > ? ? ? ?1.0.1 How do you explicitely know here that "1.0.1" is a final release ? it could be a dev snapshot of "1.0" Unless you are suggesting that snapshots are always timestamps, but that's just one way to number snapshots. [..] > If you want to then work on release 1.1, you'd continue with: > > - 1.0.567 > < 1.1a0 (which is not released and only used to mark the start of 1.1 > ? ? ? ? development, just like we do for Python) > < 1.1a0.124 > < 1.1a0.245 > < 1.1a1 ? ? (pre-release) > < 1.1a1.346 > < 1.1a2 ? ? (pre-release) > < 1.1 ? ? ? (release) > > As you can see, you don't need any dev-markers and post-markers > turn into pre-release minor version numbers... less noise, > more clarity, well at least IMHO. I see your point but the problem I see is that you are unable to explicitely make a difference between development snapshots and final releases, because projects can use three levels for their final releases: MAJOR or ?MAJOR.MINOR or MAJOR.MINOR.MICRO so if snapshots are only numbers, they can't be explicitely differenciated. IOW: 1.0.10 can be two things here, if I get it right: - a snapshot release of 1.0 - the 1.0.10 final release So end-users and packagers will not know if they deal with a final release or not. Regards Tarek From mal at egenix.com Thu Dec 3 17:19:58 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 03 Dec 2009 17:19:58 +0100 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <94bdd2610912030805i52ffa5ah8f3eb5ad120f27d9@mail.gmail.com> References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com> <4B0E6FC2.5020006@egenix.com> <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> <4B17B559.1030800@egenix.com> <94bdd2610912030718j2b79414ah74758b97805d11d8@mail.gmail.com> <4B17DE5A.8030604@egenix.com> <94bdd2610912030805i52ffa5ah8f3eb5ad120f27d9@mail.gmail.com> Message-ID: <4B17E52E.8030401@egenix.com> Tarek Ziad? wrote: > [..] >> 1. whether it's release quality code >> >> 1.0.0 >> >> 2. whether it's a development snapshot >> >> 1.0.0a0.20091202 >> >> 3. whether it's working code, but still under development >> >> 1.0.0a1 >> >> 4. whether it fixes some bug that was found after a release >> >> 1.0.1 > > How do you explicitely know here that "1.0.1" is a final release ? > > it could be a dev snapshot of "1.0" No, dev snapshots of 1.0 would be marked as: 1.0.0a0.123 > Unless you are suggesting that snapshots are always timestamps, but that's > just one way to number snapshots. Snapshots can use any number format they like - as long as it's numeric, e.g. timestamps, subversion revision numbers, etc. Not hq hex revisions, I'm afraid, unless there's a standard way to express them as (monotone) numbers as well. > [..] >> If you want to then work on release 1.1, you'd continue with: >> >> - 1.0.567 >> < 1.1a0 (which is not released and only used to mark the start of 1.1 >> development, just like we do for Python) >> < 1.1a0.124 >> < 1.1a0.245 >> < 1.1a1 (pre-release) >> < 1.1a1.346 >> < 1.1a2 (pre-release) >> < 1.1 (release) >> >> As you can see, you don't need any dev-markers and post-markers >> turn into pre-release minor version numbers... less noise, >> more clarity, well at least IMHO. > > I see your point but the problem I see is that you are unable to explicitely > make a difference between development snapshots and final releases, because > projects can use three levels for their final releases: > > MAJOR or MAJOR.MINOR or MAJOR.MINOR.MICRO > > so if snapshots are only numbers, they can't be explicitely differenciated. > > IOW: 1.0.10 can be two things here, if I get it right: > > - a snapshot release of 1.0 > - the 1.0.10 final release > > So end-users and packagers will not know if they deal with a final > release or not. A snapshot will always be a version of a pre-release, so it's clear that you get a snapshot when looking at: 1.0.0a0.123 (the "a0" signals the pre-release status) OTOH, versions without pre-release marker are always release versions, e.g. 1.0.0 1.0.0.123 1.0.0.123.0.456 (with whatever meaning those added levels may have, e.g. could be build numers, branch version numbers, etc.) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dalke at dalkescientific.com Thu Dec 3 17:32:59 2009 From: dalke at dalkescientific.com (Andrew Dalke) Date: Thu, 3 Dec 2009 17:32:59 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <4B1734AD.10500@ar.media.kyoto-u.ac.jp> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <4B1734AD.10500@ar.media.kyoto-u.ac.jp> Message-ID: On Dec 3, 2009, at 4:46 AM, David Cournapeau wrote: > The way I > would do it is by having akara distutils extensions, which define in > particular a setup function and associated classes. ... > If you think this is insane, you are not alone :) Wow. That's just crazy, I say - crazy. I considered something like that, but now I see what I was thinking of wouldn't have worked. Okay, looks like I'll have to get my mucking gear on. Andrew dalke at dalkescientific.com From floris.bruynooghe at gmail.com Thu Dec 3 18:20:30 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Thu, 3 Dec 2009 17:20:30 +0000 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <4B17DE5A.8030604@egenix.com> References: <4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com> <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> <4B17B559.1030800@egenix.com> <94bdd2610912030718j2b79414ah74758b97805d11d8@mail.gmail.com> <4B17DE5A.8030604@egenix.com> Message-ID: <20091203172030.GA31328@laurie.devork> On Thu, Dec 03, 2009 at 04:50:50PM +0100, M.-A. Lemburg wrote: > The dev markers introduce an extra level of confusion, which > IMHO is not necessary. > > Let's take 1.0a0.dev123 as example, reading it from the left: > > 1.0 - ok, so this is part of a 1.0 release > 1.0a0 - oops, no, it's not actually part of a release, it's part > of a pre-release, so move back > 1.0a0.dev123 - hmm, not even that, it's part of what's going > to become a pre-release, so move even further back This is a weak argument IMHO, because your proposal doesn't change that and this is also the way people are used too anyway ("something.alpha" has been understood to come before "something" for a long time by most people). > As developer you typically want to use the version number to > tell the user a few things about your package: > > 1. whether it's release quality code > > 1.0.0 ack > 2. whether it's a development snapshot > > 1.0.0a0.20091202 or 1.0.0a0.dev20091202 or 1.0.0a0.post20091202 (or with your case of not releasing an a0 also 1.0.0a1.dev20091202). None of these are more confusing really. > 3. whether it's working code, but still under development > > 1.0.0a1 ack > 4. whether it fixes some bug that was found after a release > > 1.0.1 ack > Sorry, I probably wasn't clear enough. What I proposed looks like this: > > 1.0a0 (which is not released and only used to mark the start of 1.0 > development, just like we do for Python) > < 1.0a0.124 > < 1.0a0.245 > < 1.0a1 (pre-release) > < 1.0a1.346 > < 1.0a2 (pre-release) > < 1.0 (release) > < 1.0.567 (release) > > and the 1.0.567 is not a dev-marker (since the proposal is about > removing these ;-), but instead a post-marker without the > "post". In reality, such a post release would be a patch-level > release. Sure you can create a release process entirely with post-release style tags. But the PEP explicitly tried to cater for both styles. In the original discussion different projects using the different systems where mentioned and in the end it was found that it was not that strange, contradictory or harming to support both. As I understand it you can follow the release process that you like already with PEP 386, and that is what is important: the proposal is capable to express the versions you desire. You can happily ignore .devXXX and others can happily use it. The only problem you then have left is that you can type .postXXX but achieve the same with just adding .XXX to the number already (as you prefer). While it is true that you could get rid of .postXXX without losing the ability to express some version numbers it was felt (all of this from my memory from original discussions - apologies if I misrepresent something) that the symmetry of .postXXX and .devXXX are actually nice. Lastly the sin of having two ways of specifying post-release tags is eased by them having a strictly defined order. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From sridharr at activestate.com Thu Dec 3 18:53:48 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 03 Dec 2009 09:53:48 -0800 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature Message-ID: <4B17FB2C.6030606@activestate.com> I thought this list might be of interest to the people here. Surprisingly not many packages rely on (not declare) setuptools' "extras" feature [bit.ly/setuptools-extras]. http://paste.pocoo.org/raw/154703/ -srid From kiorky at cryptelium.net Thu Dec 3 19:05:54 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 03 Dec 2009 19:05:54 +0100 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <4B17FB2C.6030606@activestate.com> References: <4B17FB2C.6030606@activestate.com> Message-ID: <4B17FE02.1010602@cryptelium.net> I know at least one package not in your list: zope.component [1]. [1] - http://svn.zope.org/zope.component/trunk/setup.py?rev=105736&view=markup Sridhar Ratnakumar a ?crit : > I thought this list might be of interest to the people here. > Surprisingly not many packages rely on (not declare) setuptools' > "extras" feature [bit.ly/setuptools-extras]. > > http://paste.pocoo.org/raw/154703/ > > -srid > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From fdrake at acm.org Thu Dec 3 19:10:42 2009 From: fdrake at acm.org (Fred Drake) Date: Thu, 3 Dec 2009 13:10:42 -0500 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <4B17FE02.1010602@cryptelium.net> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> Message-ID: <9cee7ab80912031010g33b724b3i6964514e55276210@mail.gmail.com> On Thu, Dec 3, 2009 at 1:05 PM, kiorky wrote: > I know at least one package not in your list: zope.component [1]. Indeed. Many z3c.* and zope.* packages provide extras that aren't referenced by other packages directly, but are referenced from buildout configurations or similar. It would be interesting to have a good way to judge the uptake for this feature, but I'm afraid this measure says little beyond "they are used." -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From attilaolah at gmail.com Thu Dec 3 19:11:26 2009 From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=) Date: Thu, 3 Dec 2009 19:11:26 +0100 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <4B17FE02.1010602@cryptelium.net> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> Message-ID: <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> Hi, On Thu, Dec 3, 2009 at 19:05, kiorky wrote: > I know at least one package not in your list: zope.component [1]. It does have a few extras (zcml being the most notable one probably), but I don't think it depends on any extras. Other packages may, however, depend on zope.component [zcml]. > [1] - http://svn.zope.org/zope.component/trunk/setup.py?rev=105736&view=markup > > Sridhar Ratnakumar a ?crit : >> I thought this list might be of interest to the people here. >> Surprisingly not many packages rely on (not declare) setuptools' >> "extras" feature [bit.ly/setuptools-extras]. >> >> ? http://paste.pocoo.org/raw/154703/ >> >> -srid >> _______________________________________________ >> Distutils-SIG maillist ?- ?Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > -- > Cordialement, > KiOrKY > GPG Key FingerPrint: 0x1A1194B7681112AF > > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From kiorky at cryptelium.net Thu Dec 3 19:16:25 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 03 Dec 2009 19:16:25 +0100 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> Message-ID: <4B180079.3060809@cryptelium.net> Attila Ol?h a ?crit : > It does have a few extras (zcml being the most notable one probably), > but I don't think it depends on any extras. Other packages may, > however, depend on zope.component [zcml]. There are many, and as i say, and i know many others packages using extras which are not yet in the list. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From sridharr at activestate.com Thu Dec 3 19:21:37 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 03 Dec 2009 10:21:37 -0800 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <4B180079.3060809@cryptelium.net> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> <4B180079.3060809@cryptelium.net> Message-ID: <4B1801B1.1030704@activestate.com> On 12/3/2009 10:16 AM, kiorky wrote: > Attila Ol?h a ?crit : >> > It does have a few extras (zcml being the most notable one probably), >> > but I don't think it depends on any extras. Other packages may, >> > however, depend on zope.component [zcml]. > There are many, and as i say, and i know many others packages using extras which > are not yet in the list. Do you have anything particular in mind? What I did was a ``grep "^.*[a-zA-Z]\["`` in the requires.txt of (almost) all the packages in PyPI. -srid From pje at telecommunity.com Thu Dec 3 19:58:46 2009 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 03 Dec 2009 13:58:46 -0500 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> Message-ID: <20091203185857.9121C3A407F@sparrow.telecommunity.com> At 01:06 AM 12/3/2009 +0100, Andrew Dalke wrote: >Hi all, > > I'm working with the Akara project. It contains a web server. The > server loads extensions from a special directory (let's say > "$AKARA" for now). An extension can register handlers for URLs. An > example extension might look like: > >installs to $AKARA/spam_extension.py >(note: only .py files are supported; not even .pyc files) >========================= >from akara.services import simple_service > >import my_spam # This is part of the distribution, and gets put in >site-packages > >@simple_service("GET", "http://vikings.protocol.id/") >def vikings(say=my_spam.DEFAULT_TEXT): > return my_spam.vikings(say) >========================= > > >We want people to be able to distribute Akara plugins and install >via setup.py. Ideally I would like to say: > >from distutils.core import setup >from akara.distutils ... I'm not sure what here ... > >setup(name="Spam services", > package="my_spam", > akara_extensions=["spam_extension.py"] >) > > >To clarify, the development/distribution package looks like: > $PACKAGE/setup.py > $PACKAGE/README > $PACKAGE/spam_extensions.py > $PACKAGE/my_spam/__init__.py > $PACKAGE/my_spam/dramatis_personae.py > $PACKAGE/my_spam/cafe.py > >and $PACKAGE/spam_extensions.py goes to $AKARA/spam_extensions.py >while $PACKAGE/my_spam is copied to site-packages. > >The installation does not need to byte-compile spam_extension.py. > >It should also include spam_extension.py in any distribution that it makes. > >I looked through the documentation and searched for existing >examples, but found nothing which does this. The plugins I found >used entry_points, and that's an architecture change which I don't >think is appropriate for us. It wouldn't be so much of a change as an addition. You'd just add code like this, either before or after your existing loop over the extensions directory: for entry_point in pkg_resources.iter_entry_points('akara'): extension_module = entry_point.load() # do stuff with extension_module And then users would declare their extensions for installation like this: setup(name="Spam services", packages=["my_spam"], py_modules=["spam_extension"], entry_points={'akara':'Spam services=spam_extension'} # arbitrary name=module ) Everything else would be the same as you described above with respect to layout, except: 1. the spam_extension module would be installed in site-packages 2. It wouldn't need to be a top-level module (i.e., it could be a module in the package) 3. You don't need any custom distutils extensions, except what you get via setuptools or Distribute. From pje at telecommunity.com Thu Dec 3 20:05:08 2009 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 03 Dec 2009 14:05:08 -0500 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <4B1801B1.1030704@activestate.com> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> <4B180079.3060809@cryptelium.net> <4B1801B1.1030704@activestate.com> Message-ID: <20091203190519.2CBAD3A4108@sparrow.telecommunity.com> At 10:21 AM 12/3/2009 -0800, Sridhar Ratnakumar wrote: > Do you have anything particular in mind? What I did was a ``grep > "^.*[a-zA-Z]\["`` in the requires.txt of (almost) all the packages in PyPI. Do note that this won't tell you about end users' use of extras. The main use case described in the documentation for extras is allowing users to install optional support for things. That means you're more likely to see them in buildout configurations (which won't be found on PyPI), and manual installation steps (which aren't recorded anywhere). From ziade.tarek at gmail.com Thu Dec 3 20:09:03 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 3 Dec 2009 20:09:03 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> Message-ID: <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> On Thu, Dec 3, 2009 at 1:06 AM, Andrew Dalke wrote: > Hi all, > > ?I'm working with the Akara project. It contains a web server. The server loads extensions from a special directory (let's say "$AKARA" for now). An extension can register handlers for URLs. An example extension might look like: > > installs to $AKARA/spam_extension.py > (note: only .py files are supported; not even .pyc files) > ========================= > from akara.services import simple_service > > import my_spam # This is part of the distribution, and gets put in site-packages > > @simple_service("GET", "http://vikings.protocol.id/") > def vikings(say=my_spam.DEFAULT_TEXT): > ? ?return my_spam.vikings(say) > ========================= > > > We want people to be able to distribute Akara plugins and install via setup.py. Ideally I would like to say: > > from distutils.core import setup > from akara.distutils ... I'm not sure what here ... > > setup(name="Spam services", > ? ? ?package="my_spam", > ? ? ?akara_extensions=["spam_extension.py"] > ) > > > To clarify, the development/distribution package looks like: > ?$PACKAGE/setup.py > ?$PACKAGE/README > ?$PACKAGE/spam_extensions.py > ?$PACKAGE/my_spam/__init__.py > ?$PACKAGE/my_spam/dramatis_personae.py > ?$PACKAGE/my_spam/cafe.py > > and $PACKAGE/spam_extensions.py goes to $AKARA/spam_extensions.py while $PACKAGE/my_spam is copied to site-packages. > > The installation does not need to byte-compile spam_extension.py. > > It should also include spam_extension.py in any distribution that it makes. > > > I looked through the documentation and searched for existing examples, but found nothing which does this. The plugins I found used entry_points, and that's an architecture change which I don't think is appropriate for us. > > Suggestions? > What about having an explicit configuration file in Akara for plugins, where you just add extensions, exactly like mercurial does: [extensions] foo = package.spam_extension bar = spam_extension2 where "package.spam_extension" and "spam_extension2" are modules Akara would simply __import__() Meaning a plugin will be a normal project that gets installed, and then configured to be used in Akara. Tarek From attilaolah at gmail.com Thu Dec 3 20:12:23 2009 From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=) Date: Thu, 3 Dec 2009 20:12:23 +0100 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <20091203190519.2CBAD3A4108@sparrow.telecommunity.com> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> <4B180079.3060809@cryptelium.net> <4B1801B1.1030704@activestate.com> <20091203190519.2CBAD3A4108@sparrow.telecommunity.com> Message-ID: <1da06520912031112p25436808r1e29e7c67652f265@mail.gmail.com> On Thu, Dec 3, 2009 at 20:05, P.J. Eby wrote: > At 10:21 AM 12/3/2009 -0800, Sridhar Ratnakumar wrote: >> >> ?Do you have anything particular in mind? What I did was a ``grep >> "^.*[a-zA-Z]\["`` in the requires.txt of (almost) all the packages in PyPI. > > Do note that this won't tell you about end users' use of extras. > > The main use case described in the documentation for extras is allowing > users to install optional support for things. ?That means you're more likely > to see them in buildout configurations (which won't be found on PyPI), and > manual installation steps (which aren't recorded anywhere). I have to agree with that. One common example are packages that use PasteScript/PasteDeploy and different configuration for development and deployment, putting only the configuration files (etc/*.ini) in the extras, so installing a deployment-version of the package wold only require installing the "deploy" extra. Another example is putting all the test files in the "test" extra, which is almost never required as a dependency, but is used by buildbots, developers, etc. From chris at simplistix.co.uk Thu Dec 3 21:25:13 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 03 Dec 2009 20:25:13 +0000 Subject: [Distutils] specifying a requirement on multiple setuptools extras Message-ID: <4B181EA9.9050001@simplistix.co.uk> Hi All, Say I have a package, mortar, that offers multiple extras options for different data storage mechanisms, eg: sqlalchemy, zodb, simpledb If I want to use this in a project and want both the sqlalchemy and zodb backends, how do I spell this? easy_install mortar[sqlalchemy,zodb] easy_install mortar[sqlalchemy] easy_install mortar[zodb] ...or something else? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ianb at colorstudy.com Thu Dec 3 21:32:58 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 3 Dec 2009 14:32:58 -0600 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <20091203190519.2CBAD3A4108@sparrow.telecommunity.com> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> <4B180079.3060809@cryptelium.net> <4B1801B1.1030704@activestate.com> <20091203190519.2CBAD3A4108@sparrow.telecommunity.com> Message-ID: On Thu, Dec 3, 2009 at 1:05 PM, P.J. Eby wrote: > At 10:21 AM 12/3/2009 -0800, Sridhar Ratnakumar wrote: >> >> ?Do you have anything particular in mind? What I did was a ``grep >> "^.*[a-zA-Z]\["`` in the requires.txt of (almost) all the packages in PyPI. > > Do note that this won't tell you about end users' use of extras. > > The main use case described in the documentation for extras is allowing > users to install optional support for things. ?That means you're more likely > to see them in buildout configurations (which won't be found on PyPI), and > manual installation steps (which aren't recorded anywhere). Incidentally, I never got around to implementing extras in pip, and no one has asked about it so far. (Though I guess pip generally encourages people handle transitive dependencies explicitly; i.e., if you want zope.component with zcml, then install zope.component and zope.component.zcml.) -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker From attilaolah at gmail.com Thu Dec 3 21:53:05 2009 From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=) Date: Thu, 3 Dec 2009 21:53:05 +0100 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> <4B180079.3060809@cryptelium.net> <4B1801B1.1030704@activestate.com> <20091203190519.2CBAD3A4108@sparrow.telecommunity.com> Message-ID: <1da06520912031253na903b8y7004db205c33b406@mail.gmail.com> Wouldn't it require zope.component to be a namespace package? Extras allw you to add extra files (subpackages) withot making the container package a namespace-package, it seems to me. On 12/3/09, Ian Bicking wrote: > On Thu, Dec 3, 2009 at 1:05 PM, P.J. Eby wrote: >> At 10:21 AM 12/3/2009 -0800, Sridhar Ratnakumar wrote: >>> >>> ?Do you have anything particular in mind? What I did was a ``grep >>> "^.*[a-zA-Z]\["`` in the requires.txt of (almost) all the packages in >>> PyPI. >> >> Do note that this won't tell you about end users' use of extras. >> >> The main use case described in the documentation for extras is allowing >> users to install optional support for things. ?That means you're more >> likely >> to see them in buildout configurations (which won't be found on PyPI), and >> manual installation steps (which aren't recorded anywhere). > > Incidentally, I never got around to implementing extras in pip, and no > one has asked about it so far. (Though I guess pip generally > encourages people handle transitive dependencies explicitly; i.e., if > you want zope.component with zcml, then install zope.component and > zope.component.zcml.) > > > -- > Ian Bicking | http://blog.ianbicking.org | > http://topplabs.org/civichacker > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Sent from my mobile device From pje at telecommunity.com Thu Dec 3 22:20:11 2009 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 03 Dec 2009 16:20:11 -0500 Subject: [Distutils] specifying a requirement on multiple setuptools extras In-Reply-To: <4B181EA9.9050001@simplistix.co.uk> References: <4B181EA9.9050001@simplistix.co.uk> Message-ID: <20091203212023.8E7D33A407F@sparrow.telecommunity.com> At 08:25 PM 12/3/2009 +0000, Chris Withers wrote: >Hi All, > >Say I have a package, mortar, that offers multiple extras options >for different data storage mechanisms, eg: sqlalchemy, zodb, simpledb > >If I want to use this in a project and want both the sqlalchemy and >zodb backends, how do I spell this? > >easy_install mortar[sqlalchemy,zodb] Just like that. From pje at telecommunity.com Thu Dec 3 22:21:20 2009 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 03 Dec 2009 16:21:20 -0500 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <1da06520912031253na903b8y7004db205c33b406@mail.gmail.com> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> <4B180079.3060809@cryptelium.net> <4B1801B1.1030704@activestate.com> <20091203190519.2CBAD3A4108@sparrow.telecommunity.com> <1da06520912031253na903b8y7004db205c33b406@mail.gmail.com> Message-ID: <20091203212131.9B6B83A407F@sparrow.telecommunity.com> At 09:53 PM 12/3/2009 +0100, Attila Ol??h wrote: >Wouldn't it require zope.component to be a namespace package? Extras >allw you to add extra files (subpackages) withot making the container >package a namespace-package, it seems to me. No, they don't. Extras just cause a project to pull in additional dependencies, not additional *files*. From ianb at colorstudy.com Thu Dec 3 22:37:04 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 3 Dec 2009 15:37:04 -0600 Subject: [Distutils] 70 packages in total use setuptools' "extras" feature In-Reply-To: <20091203212131.9B6B83A407F@sparrow.telecommunity.com> References: <4B17FB2C.6030606@activestate.com> <4B17FE02.1010602@cryptelium.net> <1da06520912031011i7cdc8e9ue42d5ef7a5aeca44@mail.gmail.com> <4B180079.3060809@cryptelium.net> <4B1801B1.1030704@activestate.com> <20091203190519.2CBAD3A4108@sparrow.telecommunity.com> <1da06520912031253na903b8y7004db205c33b406@mail.gmail.com> <20091203212131.9B6B83A407F@sparrow.telecommunity.com> Message-ID: On Thu, Dec 3, 2009 at 3:21 PM, P.J. Eby wrote: > At 09:53 PM 12/3/2009 +0100, Attila Ol?h wrote: >> >> Wouldn't it require zope.component to be a namespace package? Extras >> allw you to add extra files (subpackages) withot making the container >> package a namespace-package, it seems to me. > > No, they don't. ?Extras just cause a project to pull in additional > dependencies, not additional *files*. The basic advantage, I guess, is the indirection; you can require zope.component[zcml], and the maintainers of zope.component determine what is necessary to provide that functionality. And from the changelog it seems they have updated that to depend on different things over time. Though I suspect that was actually a kind of misfeature of extras... some library X requires zope.component[zcml], but in fact only requires a subset of that functionality, and some framework F requires library X, which draws in zope.component[zcml], which in turn requires oodles of other things, and there is pushback from F to zope.component (maybe involving library X maintainers, or not) to reduce the dependencies to what is more strictly required. All of which requires lots of tedious communication and negotiation, which I don't think is really that helpful for anyone, and is why I prefer a more manual assembling of sets of libraries, and am not a fan of Setuptools runtime checking. My general feeling is that these dependencies should only be loosely trusted; only actual testing can confirm that an update is correct for a developer's use case. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker From chris at simplistix.co.uk Thu Dec 3 23:01:50 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 03 Dec 2009 22:01:50 +0000 Subject: [Distutils] specifying a requirement on multiple setuptools extras In-Reply-To: <20091203212023.8E7D33A407F@sparrow.telecommunity.com> References: <4B181EA9.9050001@simplistix.co.uk> <20091203212023.8E7D33A407F@sparrow.telecommunity.com> Message-ID: <4B18354E.7040900@simplistix.co.uk> P.J. Eby wrote: >> If I want to use this in a project and want both the sqlalchemy and >> zodb backends, how do I spell this? >> >> easy_install mortar[sqlalchemy,zodb] > > Just like that. Cool :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From read.beyond.data at gmx.net Fri Dec 4 05:41:09 2009 From: read.beyond.data at gmx.net (Celvin) Date: Fri, 4 Dec 2009 05:41:09 +0100 Subject: [Distutils] Enumerating packages in a distribution Message-ID: <338311894.20091204054109@gmx.net> Hi, I recently implemented a plugin system using setuptools, and now I'm wondering how to enumerate the packages contained in a given Distribution object. What I'm doing is basically this: I use egg files as self-describing plugins to allow users to distribute plugins containing several packages, with each package providing a set of functions that are registered and loaded into a GUI's menu at runtime. The way each function in a given package is displayed in the GUI is encoded into a XML file that is distributed with each package, i.e. users can add several packages to the distribution, declare the entry points and create an XML file that will describe each package's menu structure like this: foo/ foo/ __init__.py setup.py package1/ menu.xml functions1.py package2/ menu.xml functions2.py ... Accessing the XML files via pkg_resource.resource_stream(__name__, "menu.xml") works just fine from inside functionX.py, but since the combination of working_set.find_plugins and working_set.iter_entry_points returns a loose list of EntryPoint instances without preserving the package structure, I have trouble mapping an EntryPoint to the matching menu.xml file, while only parsing the XML file just once and not for each entry point. In the past, I used the module_name attribute of the EntryPoint instances to build a path to be used with the IResourceProvider API implemented in each Distribution instance like this: dist.get_resource_stream(__name__, "%s/menu.xml" % (entry_point.module_name, )) ...but since the user creating a plugin is free to nest the package further, the module_name might not be identical to the package name where the menu.xml file is stored, and I'm hesitant to just split module_name at the dots and just use the first part, as it somehow "doesn't feel right". The whole thing somehow seems overly complicated, given that I'm implementing it in Python, so maybe I'm artificially complicating things. If someone could provide some insights or simplify my plugin scheme, I would really appreciate it. Cheers, Celvin From david at ar.media.kyoto-u.ac.jp Fri Dec 4 05:29:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 04 Dec 2009 13:29:39 +0900 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> Message-ID: <4B189033.3050308@ar.media.kyoto-u.ac.jp> Tarek Ziad? wrote: > where "package.spam_extension" and "spam_extension2" are modules Akara > would simply __import__() > Meaning a plugin will be a normal project that gets installed, and > then configured to be used in Akara. > This solution is simpler, but it does not solve the issue of installing the plugin outside site-packages. I think it is good practice not to pollute site-packages with app-specific plugins. David From kiorky at cryptelium.net Fri Dec 4 09:18:05 2009 From: kiorky at cryptelium.net (kiorky) Date: Fri, 04 Dec 2009 09:18:05 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <4B189033.3050308@ar.media.kyoto-u.ac.jp> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> <4B189033.3050308@ar.media.kyoto-u.ac.jp> Message-ID: <4B18C5BD.3070108@cryptelium.net> David Cournapeau a ?crit : > Tarek Ziad? wrote: >> where "package.spam_extension" and "spam_extension2" are modules Akara >> would simply __import__() >> Meaning a plugin will be a normal project that gets installed, and >> then configured to be used in Akara. >> > > This solution is simpler, but it does not solve the issue of installing > the plugin outside site-packages. I think it is good practice not to > pollute site-packages with app-specific plugins. ZopeComponentArchitecture/Entry points with an assembler like buildout for the later won't pollute site-packages > > David > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From chris at simplistix.co.uk Fri Dec 4 09:44:51 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 04 Dec 2009 08:44:51 +0000 Subject: [Distutils] PEP 345 - 3 new fields In-Reply-To: <20091202230109.GB15234@laurie.devork> References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com> <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com> <94bdd2610912010711v6769cedxa1b5352d35f7c598@mail.gmail.com> <94bdd2610912010750x20c02430s72c6ba22b8c0b08a@mail.gmail.com> <4B156CE5.4080200@trueblade.com> <94bdd2610912020643x4979021ete9313642176d2113@mail.gmail.com> <20091202230109.GB15234@laurie.devork> Message-ID: <4B18CC03.9050903@simplistix.co.uk> Floris Bruynooghe wrote: > On Wed, Dec 02, 2009 at 03:43:28PM +0100, Tarek Ziad? wrote: >> Ok so, if everyone agrees on those two, I am proposing to add for now : >> >> - Repository-Browse-URL : url to a web-browseable repository (no >> distinction whether its the trunk, >> a branch etc) > > I'm still uneasy about forcing everyone to have a browsable URL and > would rather just have a Repository-URL which is reccomended to be > browsable but doens't have to. But if I'm in the minority then so be > it. Who said anything about forcing? All of these extra fields are optional... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Dec 4 09:48:17 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 04 Dec 2009 08:48:17 +0000 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <4B189033.3050308@ar.media.kyoto-u.ac.jp> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> <4B189033.3050308@ar.media.kyoto-u.ac.jp> Message-ID: <4B18CCD1.6040107@simplistix.co.uk> David Cournapeau wrote: > This solution is simpler, but it does not solve the issue of installing > the plugin outside site-packages. I think it is good practice not to > pollute site-packages with app-specific plugins. Just use buildout and be done with it ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From denis-bz-py at t-online.de Fri Dec 4 11:43:51 2009 From: denis-bz-py at t-online.de (denis) Date: Fri, 04 Dec 2009 11:43:51 +0100 Subject: [Distutils] setuptools-0.6c11 package_index.py 475 return dist.clone ? Message-ID: Folks, in setuptools-0.6c11-py2.6.egg/setuptools/package_index.py lines 468-475 if dist is None: ... return dist.clone(...) => of course AttributeError: 'NoneType' object has no attribute 'clone' return None if dist is None else dist.clone(...) fixes yolk. (Does nobody else use yolk ? I like it, less is more; which of this year's new package managers have the same functionality ?) Sorry if this is a duplicate or the wrong place to post. cheers -- denis From aljosa.mohorovic at gmail.com Fri Dec 4 15:42:21 2009 From: aljosa.mohorovic at gmail.com (=?UTF-8?B?QWxqb8WhYSBNb2hvcm92acSH?=) Date: Fri, 4 Dec 2009 15:42:21 +0100 Subject: [Distutils] install_requires option to force pypi url? Message-ID: <87d364ab0912040642r71b3ab62k9ea76f05d40eeb3c@mail.gmail.com> i've setup private pypi where i can upload some packages that are not suitable for pypi.python.org. my package, let's call it myapp, contains "install_requires=['django',]" but with that requirement i can't install it. -------------------------------------------------- $ virtualenv --no-site-packages --distribute env New python executable in env/bin/python A globally installed setuptools was found (in /usr/lib/python2.6/dist-packages) Use the --no-site-packages option to use distribute in the virtualenv. Installing distribute...........................................................................................................................................................................done. $ . env/bin/activate $ pip freeze distribute==0.6.8 wsgiref==0.1.2 $ pip install myapp -i http://pypi.example.com Downloading/unpacking myapp Using download cache from /home/aljosa/.pip-cache/ Running setup.py egg_info for package myapp Downloading/unpacking django (from myapp) Could not find any downloads that satisfy the requirement django (from myapp) No distributions at all found for django (from myapp) -------------------------------------------------- since i can normally install django with "pip install django" i assume that when i set "-i http://pypi.example.com" it's also used for install_requires. any tips on using multiple pypi repositories? is there some way to define which pypi url to use for package installation? Aljosa Mohorovic From sridharr at activestate.com Fri Dec 4 19:04:54 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Fri, 04 Dec 2009 10:04:54 -0800 Subject: [Distutils] setuptools-0.6c11 package_index.py 475 return dist.clone ? In-Reply-To: References: Message-ID: <4B194F46.9050206@activestate.com> On 12/4/2009 2:43 AM, denis wrote: > Folks, > in setuptools-0.6c11-py2.6.egg/setuptools/package_index.py lines 468-475 > > if dist is None: > ... > return dist.clone(...) > > => of course > AttributeError: 'NoneType' object has no attribute 'clone' Fixed in setuptools trunk - http://bugs.python.org/setuptools/issue90 > return None if dist is None else dist.clone(...) > fixes yolk. > (Does nobody else use yolk ? I like it, less is more; > which of this year's new package managers have the same functionality Does Yolk use setuptools? If so, you may try porting it to Distribute: http://python-distribute.org/ -srid From ziade.tarek at gmail.com Fri Dec 4 19:39:58 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 4 Dec 2009 19:39:58 +0100 Subject: [Distutils] setuptools-0.6c11 package_index.py 475 return dist.clone ? In-Reply-To: <4B194F46.9050206@activestate.com> References: <4B194F46.9050206@activestate.com> Message-ID: <94bdd2610912041039x2268f2c3t368ef8b437fd2ee@mail.gmail.com> On Fri, Dec 4, 2009 at 7:04 PM, Sridhar Ratnakumar wrote: > On 12/4/2009 2:43 AM, denis wrote: >> >> Folks, >> ? in setuptools-0.6c11-py2.6.egg/setuptools/package_index.py lines 468-475 >> >> if dist is None: >> ... >> return dist.clone(...) >> >> => of course >> AttributeError: 'NoneType' object has no attribute 'clone' > > Fixed in setuptools trunk - http://bugs.python.org/setuptools/issue90 Notice that this should work fine in Distribute. (the bug is from a change in setuptools we didn't mirror) > >> return None if dist is None else dist.clone(...) >> fixes yolk. >> (Does nobody else use yolk ? I like it, less is more; >> which of this year's new package managers have the same functionality > > Does Yolk use setuptools? If so, you may try porting it to Distribute: > http://python-distribute.org/ > > -srid > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From dalke at dalkescientific.com Sat Dec 5 00:22:45 2009 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 5 Dec 2009 00:22:45 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> Message-ID: <91053537-54AF-4A22-9838-EEB46E9672FE@dalkescientific.com> On Dec 3, 2009, at 8:09 PM, Tarek Ziad? wrote: > What about having an explicit configuration file in Akara for plugins, > where you just add extensions, exactly like mercurial does: > > [extensions] > foo = package.spam_extension > bar = spam_extension2 > > where "package.spam_extension" and "spam_extension2" are modules Akara > would simply __import__() > Meaning a plugin will be a normal project that gets installed, and > then configured to be used in Akara. Since I don't know how mercurial does this, I'm going to have to guess on your meaning here and reading the hg documentation. There seem to be two ways to install an extension. One is by editing the hgrc file, and the other is by putting the Python code in the "hgext" module. I looked at the installation instructions for mercurial packages listed at http://mercurial.selenic.com/wiki/UsingExtensions . They all require hand-editing of the hgrc file. That is not what I want. What I want is "python setup.py install" to work, and to allow third-party extensions to use the existing Python setup framework to install not only the extension component, but any other packages and even compiled C code which might be needed. The other option was to put files into hgext, but that seems only appropriate for the extensions shipped with Mercurial, and I found nothing which helped automating the installation of those files. So your recommendation does not seem helpful. Andrew dalke at dalkescientific.com From dalke at dalkescientific.com Sat Dec 5 00:34:11 2009 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 5 Dec 2009 00:34:11 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <4B18C5BD.3070108@cryptelium.net> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> <4B189033.3050308@ar.media.kyoto-u.ac.jp> <4B18C5BD.3070108@cryptelium.net> Message-ID: <31DAE64C-0406-48C0-9C16-FF58C22037AE@dalkescientific.com> On Dec 4, 2009, at 9:18 AM, kiorky wrote: > ZopeComponentArchitecture/Entry points with an assembler like buildout for the > later won't pollute site-packages I tried searching for "ZopeComponentArchitecture/Entry points" but found nothing that seemed relevant. Did you mean "setuptools entry_points"? I only know a bit about them, in the context of using TurboGears, but have not used them in my own code. I had some concerns about them. For one, all of the plugins define out-word facing web services on our server. If the plugins can be located in arbitrary locations inside of site-packages, how does the administrator know which plugins will be activated? How does the admin enable/disable a plugin for testing or security reasons, except by changing the entire package installation? If the admin wants to change the URL for a given service from "/spam" to "/spam_and_eggs", which is currently done as configuration data in the installed plugin, file, that is, by changing # default service name is base on the function name. This becomes "/spam" @simple_service("GET", "http://protocol.id/") def spam(): return "Spam!" -to- @simple_service("GET", "http://protocol.id/", "spam_and_eggs") def spam(): return "Spam!" It does not seem like changing the installed package will be so simple. It would mean searching the installed site-packages to find "spam" (instead of now, which is done with a grep of a single directory) and possibly made more difficult if the package is installed as an egg. These are reasons I decided not to look into setuptools' entry_points as a solution to this problem. Andrew dalke at dalkescientific.com From dalke at dalkescientific.com Sat Dec 5 00:44:29 2009 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 5 Dec 2009 00:44:29 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <4B189033.3050308@ar.media.kyoto-u.ac.jp> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> <4B189033.3050308@ar.media.kyoto-u.ac.jp> Message-ID: On Dec 4, 2009, at 5:29 AM, David Cournapeau wrote: > This solution is simpler, but it does not solve the issue of installing > the plugin outside site-packages. I think it is good practice not to > pollute site-packages with app-specific plugins. This tangential topic came up recently because of our recent GothPy (local Gothenburg, Sweden Python User's Group) meeting. One of the members develops TextTest http://texttest.carmen.se/index.php?page=main It is a command-line application with both GUI and non-GUI modes. The GUI uses Gtk. He does not distribute it with setup.py . The installation instructions are "call $texttest/bin/texttest.py" or set your path to include the right directory. This causes a problem with me because my system Python is not configured for PyGtk. I have a Mac. I used Macports to install PyGtk and use the Python in /opt/local/bin/python2.6 . I ended up putting that path in by-hand. I would rather have had the normal setup.py script installation program take care of that for me. However, he does not want to install it as a package because the code is not designed as a library, and he doesn't want to "pollute" site-packages either with this application code. I said it was okay, and that people write command-line programs like this #!/usr/bin/env python import application_package application_package.main() I thought that was common practice, and I don't see it as pollution. But now I see my views aren't as much in the majority as I thought they were. What should the practice be? Put this into "/usr/local/lib/application_package-$version/ and have the command-line program look more like #!/usr/bin/env python import sys sys.path.insert(0, "/usr/local/lib/application-package-$version") import application_package application_package.main() ? Andrew dalke at dalkescientific.com From dalke at dalkescientific.com Sat Dec 5 01:18:08 2009 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 5 Dec 2009 01:18:08 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <20091203185857.9121C3A407F@sparrow.telecommunity.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <20091203185857.9121C3A407F@sparrow.telecommunity.com> Message-ID: <4DFA5512-54F4-4FE1-8F3D-2FE1E90A4CDA@dalkescientific.com> On Dec 3, 2009, at 7:58 PM, P.J. Eby wrote: > It wouldn't be so much of a change as an addition. You'd just add code like this, either before or after your existing loop over the extensions directory: > > for entry_point in pkg_resources.iter_entry_points('akara'): > extension_module = entry_point.load() > # do stuff with extension_module I sent a followup to Tarek's reply on some of the architectural changes I meant, like changing from the current ability to identify/enable/disable/rename web services with text editor to one which I don't know much about and which seems more complicated. While thinking about this, there's another one I came up with. We use a pre-forking server based on flup. The master starts up and spawns the processes which do the listening. It's these processes which import the extensions. Send a SIGHUP to the master and it restarts the children, which in turn rescan the list of extensions and reimport them. It appears that pkg_resources does some caching, including using linecache. I can't tell how well it would work if our pluging packages were updated after the main server was running. Andrew dalke at dalkescientific.com From ziade.tarek at gmail.com Sat Dec 5 01:44:58 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 5 Dec 2009 01:44:58 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <91053537-54AF-4A22-9838-EEB46E9672FE@dalkescientific.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> <91053537-54AF-4A22-9838-EEB46E9672FE@dalkescientific.com> Message-ID: <94bdd2610912041644x49a5b43fnb4ff1c6680e74d2a@mail.gmail.com> On Sat, Dec 5, 2009 at 12:22 AM, Andrew Dalke wrote: [..] > I looked at the installation instructions for mercurial packages listed at http://mercurial.selenic.com/wiki/UsingExtensions . They all require hand-editing of the hgrc file. That is not what I want. Yes, that's how it works, you can add in ~/.hgrc: [extensions] package.spam_extension = ... And mercurial will enventually do an __import__("package.spam_extension"). That's an explicit plugin system. Setuptools entry_points are a plugin system based on discovery: you register some code and you can iterate on all registered code from your app. In the next mail you are writing about entry_points this: > I had some concerns about them. For one, all of the plugins define out-word facing web services on > our server. If the plugins can be located in arbitrary locations inside of site-packages, how does > the administrator know which plugins will be activated? How does the admin enable/disable a > plugin for testing or security reasons, except by changing the entire package installation? This means that you want to be able to *manage* and configure your plugins, so unless you are driving them from an explicit central configuration system like mercurial has, you won't be able to fully control plugins, which can lead to the limitations you are describing. The other way is to force the plugins to be located in the same directory as you said, but this means that you want to allow any third-party application to install itself as a plugin in there. Then if the administrator wants to deactivate some plugins, what will he do ? remove the file that was added in the special directory ? That could make that particular plugin half-removed if it has other files in the system. IOW if a plugin is installed in the system, it has to be fully installed within your application if you want full control over it. That's why I was thinking of a configuration file, where it is easy to control your plugins explicitely. Of course, until we have the APIs (PEP 376) to uninstall packages easily, it's not easy to remove an installed project that may contain the plugin, unless it was installed with Pip. Or, maybe you could use self-contained eggs, like zc.buildout does, that might be the only use case for them Regards Tarek From dalke at dalkescientific.com Sat Dec 5 02:27:10 2009 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 5 Dec 2009 02:27:10 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <4B1734AD.10500@ar.media.kyoto-u.ac.jp> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <4B1734AD.10500@ar.media.kyoto-u.ac.jp> Message-ID: On Dec 3, 2009, at 4:46 AM, David Cournapeau wrote: > If you think this is insane, you are not alone :) I think that by hooking into "data_files" I can be a bit less insane. What's wrong with this? Other than that it writes Writing /Library/Python/2.6/site-packages/Spam_services-0.0.0-py2.6.egg-info which isn't needed when there is no other Python code. But I can special case that and bypass the setup mechanism entirely if there are no py_modules, packages, or ext_modules given. (Am I missing something?) from distutils.core import setup def my_setup(**kwargs): if "akara_extensions" in kwargs: akara_extensions = kwargs["akara_extensions"] if not isinstance(akara_extensions, list): raise TypeError("akara_extensions must be a list of filenames") del kwargs["akara_extensions"] if "data_files" not in kwargs: kwargs["data_files"] = [] data_files = kwargs["data_files"] data_files.append( ("/tmp", akara_extensions) ) setup(**kwargs) my_setup(name="Spam services", #packages=["my_spam"], akara_extensions = ["spam_extensions.py"], ) Andrew dalke at dalkescientific.com From cournape at gmail.com Sat Dec 5 03:10:26 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 5 Dec 2009 11:10:26 +0900 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <4B1734AD.10500@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220912041810m76777661ke528dc0f4478e27b@mail.gmail.com> On Sat, Dec 5, 2009 at 10:27 AM, Andrew Dalke wrote: > On Dec 3, 2009, at 4:46 AM, David Cournapeau wrote: >> If you think this is insane, you are not alone :) > > I think that by hooking into "data_files" I can be a bit less insane. What's wrong with this? The problem of data_files is that its semantics are heavily changed between the different tools (distutils, setuptools, etc...), so covering all cases is difficult. From my experience, creating a new command makes the thing more robust against monkey patching (don't expect too much, though). David From ziade.tarek at gmail.com Sat Dec 5 03:49:13 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 5 Dec 2009 03:49:13 +0100 Subject: [Distutils] RFC822, PKG-INFO and the Description field Message-ID: <94bdd2610912041849m1b140b8fu8a3faf1f6f478ff4@mail.gmail.com> Hi, I am currently fixing a bug in Distutils: http://bugs.python.org/issue1923 this bugs makes a Description field like: """ Text:: a literal python block:: >>> import this """ Transformed into : """ Text:: a literal python block:: >>> import this """ Which is fine for RFC822 compliancy but sucks for reST if someone wants to parse it back. There's another problem: empty lines. For instance: """ Description: Text:: a literal python block:: >>> import this """ Will not be parseable with rfc822 because the first empty line ends the header. IOW we need to encode that multi-line field differently. I want to take the chance that we are changing PEP 345, to introduce a smarter marker in 1.2, that will add a character (:) after the 8 spaces to avoid losing empty lines: """ Description: Text:: : :a literal python block:: : : >>> import this """ So we are able to unparse it. Thoughts ? Regards Tarek From pje at telecommunity.com Sat Dec 5 05:26:14 2009 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 04 Dec 2009 23:26:14 -0500 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <31DAE64C-0406-48C0-9C16-FF58C22037AE@dalkescientific.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> <4B189033.3050308@ar.media.kyoto-u.ac.jp> <4B18C5BD.3070108@cryptelium.net> <31DAE64C-0406-48C0-9C16-FF58C22037AE@dalkescientific.com> Message-ID: <20091205042628.275EB3A408B@sparrow.telecommunity.com> At 12:34 AM 12/5/2009 +0100, Andrew Dalke wrote: >I had some concerns about them. For one, all of the plugins define >out-word facing web services on our server. If the plugins can be >located in arbitrary locations inside of site-packages, how does the >administrator know which plugins will be activated? How does the >admin enable/disable a plugin for testing or security reasons, >except by changing the entire package installation? Then install the plugins as eggs to the plugin directory, rather than installing them in site-packages. >If the admin wants to change the URL for a given service from >"/spam" to "/spam_and_eggs", which is currently done as >configuration data in the installed plugin, file, that is, by changing > ># default service name is base on the function name. This becomes "/spam" >@simple_service("GET", "http://protocol.id/") >def spam(): > return "Spam!" > > -to- > >@simple_service("GET", "http://protocol.id/", "spam_and_eggs") >def spam(): > return "Spam!" > > >It does not seem like changing the installed package will be so simple. Indeed. It would be much better to make your service decorator read overrides from a configuration file, so that the decorator values are merely defaults. Editing source code is a lousy way to do configuration, when there's stuff in the file besides configuration. From pje at telecommunity.com Sat Dec 5 05:30:35 2009 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 04 Dec 2009 23:30:35 -0500 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <4DFA5512-54F4-4FE1-8F3D-2FE1E90A4CDA@dalkescientific.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <20091203185857.9121C3A407F@sparrow.telecommunity.com> <4DFA5512-54F4-4FE1-8F3D-2FE1E90A4CDA@dalkescientific.com> Message-ID: <20091205043048.C77D73A408B@sparrow.telecommunity.com> At 01:18 AM 12/5/2009 +0100, Andrew Dalke wrote: >It appears that pkg_resources does some caching, including using >linecache. I can't tell how well it would work if our pluging >packages were updated after the main server was running. If you need to ensure that you get a fresh list of plugins each time, you can use "for entry_point in pkg_resources.WorkingSet().iter_entry_points(...)". Or more precisely, you can create a new WorkingSet() whenever you want to start over with a clean cache. (Btw, the only thing pkg_resources uses linecache for is to ensure that the source of a script loaded from an alternate location is viewable in linecache as though it came from the location the script was run from. This is a debugging aid only, and doesn't have any effect on anything that's not trying to dump out source code lines (such as an error reporting tool, or the Python debugger).) From kiorky at cryptelium.net Sat Dec 5 11:06:02 2009 From: kiorky at cryptelium.net (kiorky) Date: Sat, 05 Dec 2009 11:06:02 +0100 Subject: [Distutils] installing .py plugins to an alternate directory In-Reply-To: <31DAE64C-0406-48C0-9C16-FF58C22037AE@dalkescientific.com> References: <9AC0CDDA-76A9-4691-8B3C-86CD0A7DB4B9@dalkescientific.com> <94bdd2610912031109k51b35e80h5fc7d48e99d859a2@mail.gmail.com> <4B189033.3050308@ar.media.kyoto-u.ac.jp> <4B18C5BD.3070108@cryptelium.net> <31DAE64C-0406-48C0-9C16-FF58C22037AE@dalkescientific.com> Message-ID: <4B1A308A.8070803@cryptelium.net> Andrew Dalke a ?crit : > On Dec 4, 2009, at 9:18 AM, kiorky wrote: >> ZopeComponentArchitecture/Entry points with an assembler like buildout for the >> later won't pollute site-packages > > > I tried searching for "ZopeComponentArchitecture/Entry points" but found nothing that seemed relevant. Did you mean "setuptools entry_points"? I mean two approaches: - setuptools entry points, if you need just simple entry points registered when your plugins are in the sys.path of your package. So when you have generated scripts with buildout, or when you have done some easy_install or python setup.py install dance. - Zope Component Architecture, if you need something more complicated. The later can be tedious at first if you are not yet familiar with zope as you ll surely have to initiate a 'site' yourself and do other stuff around if you are not in a zope environment, but that is just a matter of some readings and hacks. Starters are the zope.interface, zope.configuration, and zope.component pypi pages ([1]) and also this one [2]. Note that Trac do something similar on it's own if i could remember well. [1] - http://pypi.python.org/pypi/zope.configuration http://pypi.python.org/pypi/zope.interface http://pypi.python.org/pypi/zope.component [2] - http://www.muthukadan.net/docs/zca.html > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From Ronny.Pfannschmidt at gmx.de Sun Dec 6 17:37:38 2009 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Sun, 06 Dec 2009 17:37:38 +0100 Subject: [Distutils] RFC822, PKG-INFO and the Description field In-Reply-To: <94bdd2610912041849m1b140b8fu8a3faf1f6f478ff4@mail.gmail.com> References: <94bdd2610912041849m1b140b8fu8a3faf1f6f478ff4@mail.gmail.com> Message-ID: <1260117458.6526.28.camel@localhost> On Sat, 2009-12-05 at 03:49 +0100, Tarek Ziad? wrote: > Hi, > > I am currently fixing a bug in Distutils: http://bugs.python.org/issue1923 > > this bugs makes a Description field like: > > """ > Text:: > a literal python block:: > >>> import this > """ > > Transformed into : > > """ > Text:: > a literal python block:: > >>> import this > """ > > Which is fine for RFC822 compliancy but sucks for reST if someone > wants to parse it back. > > There's another problem: empty lines. > > For instance: > > """ > Description: Text:: > > a literal python block:: > > >>> import this > """ > > Will not be parseable with rfc822 because the first empty line ends the header. > > IOW we need to encode that multi-line field differently. > > I want to take the chance that we are changing PEP 345, to introduce a > smarter marker > in 1.2, that will add a character (:) after the 8 spaces to avoid > losing empty lines: > > """ > Description: Text:: > : > :a literal python block:: > : > : >>> import this > """ > > So we are able to unparse it. > > Thoughts ? How about turning that file into a real mime message instead of just a set of pseudo mime headers with some pseudo encodings for multiline stuff. That way the long description could be the body of that message, no more messy recoding needed. As far as i can tell, all the other optional fields are designed to fit single lines anyway. IMHO its a perfect match. If the idea is liked, i'd take 3-4 Hours to write a new DistributionMetadata class that handles the new format as well as backward compatibility (i suppose a new version number is needed anyway for some of the other additions) Regards Ronny From ziade.tarek at gmail.com Sun Dec 6 21:22:32 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 6 Dec 2009 21:22:32 +0100 Subject: [Distutils] RFC822, PKG-INFO and the Description field In-Reply-To: <1260117458.6526.28.camel@localhost> References: <94bdd2610912041849m1b140b8fu8a3faf1f6f478ff4@mail.gmail.com> <1260117458.6526.28.camel@localhost> Message-ID: <94bdd2610912061222h49eb0cb4te8887b88d509bb25@mail.gmail.com> On Sun, Dec 6, 2009 at 5:37 PM, Ronny Pfannschmidt wrote: [..] > > How about turning that file into a real mime message instead of just a > set of pseudo mime headers with some pseudo encodings for multiline > stuff. > > That way the long description could be the body of that message, > no more messy recoding needed. > > As far as i can tell, all the other optional fields are designed to fit > single lines anyway. > The problem is, we may have in the future more multi-line fields so I think we should not use a message-like body. It's not a big problem if PKG-INFO looks messy, as long as we provide APIs to write *and* read it. Notice that there is not official consumers for this file yet as far as I know, PyPI reads the metadata from a form that is pushed by the register command and stores it in its own format. OTHO we could drop RFC 822 completely and go for a simpler format like json for 1.2.. I am not sure how many third party applications this will impact but I don't think there are so many. In any case, they will have to adapt themselves for 1.2, so maybe keeping a RFC 822-like format is less annoying for them. Regards Tarek From ziade.tarek at gmail.com Mon Dec 7 21:35:09 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 7 Dec 2009 21:35:09 +0100 Subject: [Distutils] PEP 386 status - last round here ? In-Reply-To: <4B17E52E.8030401@egenix.com> References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com> <4B1175D8.4090107@egenix.com> <20091128211007.311B93A4119@sparrow.telecommunity.com> <4B127852.7010908@egenix.com> <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com> <4B17B559.1030800@egenix.com> <94bdd2610912030718j2b79414ah74758b97805d11d8@mail.gmail.com> <4B17DE5A.8030604@egenix.com> <94bdd2610912030805i52ffa5ah8f3eb5ad120f27d9@mail.gmail.com> <4B17E52E.8030401@egenix.com> Message-ID: <94bdd2610912071235k2ad922f6wc43f469eb5fcf62f@mail.gmail.com> On Thu, Dec 3, 2009 at 5:19 PM, M.-A. Lemburg wrote: [..] > > A snapshot will always be a version of a pre-release, so > it's clear that you get a snapshot when looking at: > > ? ? ? ?1.0.0a0.123 > > (the "a0" signals the pre-release status) > > OTOH, versions without pre-release marker are always release > versions, e.g. > > ? ? ? ?1.0.0 > ? ? ? ?1.0.0.123 > ? ? ? ?1.0.0.123.0.456 > > (with whatever meaning those added levels may have, e.g. could be > build numers, branch version numbers, etc.) I am with Floris here, I think dev/post markers are more explicit than "a0" and implicit markers. And we will always have a hundred ways of doing this. Last, as long as people schemes can be translated to PEP 386, wer'e safe. I propose to end this thread here, and to start a new thread on Python-dev for PEP 386 as I mentioned previously. So, eventually we can decide there on a final scheme, then focus on the last details on PEP 345. I will post the mail for python-dev here so people can check it before it is sent, Regards Tarek From ziade.tarek at gmail.com Mon Dec 7 22:53:11 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 7 Dec 2009 22:53:11 +0100 Subject: [Distutils] Proposing PEP 386 - a standard version scheme Message-ID: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> (This is the email I will send to python-dev, so everyone can review it before it is sent) On behalf of the Distutils-SIG, I would like to propose PEP 386 for inclusion in the sdtlib, and have a final discussion here on Python-dev. http://www.python.org/dev/peps/pep-0386 This PEP has been discussed for some time in Distutils-SIG, and we agreed there that it's important to have it accepted for the future of packaging because: 1/ it will offer interoperability for all Python package managers out there. (namely: Distutils, Distribute, Setuptools, Pip, PyPM) 2/ it will provide a reference for OS packagers, so they can translate automatically Python distributions versions in their own schemes. Choosing a schema was quite controversial because every development team have their own version scheme. But notice that it is not in the scope of this PEP to come up with an universal versioning schema, intended to support many or all existing versioning schemas. There will always be competing grammars, either mandated by distro or project policies or by historical reasons and we cannot expect that to change. The goal of this PEP is rather to provide a standard reference schema that is able to express the usual versioning semantics, so it's possible to parse any alternative versioning schema and transform it into a compliant one. This is how OS packagers usually deal with the existing version schemas and is a preferable alternative than supporting an arbitrary set of versioning schemas. As I expected, we didn't find a full consensus on the final PEP 386 schema on Distutils-SIG. But the one presented is good enough for what we need to express, as far as I am concerned (and some other people). This status understandable, and can go on for months in distutils-SIG. So I am proposing to see this PEP discussed for one more round here, then rejected or approved by the highest authority :) so we can move on to the final changes we are planning on PEP 345 and PEP 376 (those are the next PEP we want to propose here) Regards Tarek From dsdale24 at gmail.com Mon Dec 7 23:16:33 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 7 Dec 2009 17:16:33 -0500 Subject: [Distutils] Proposing PEP 386 - a standard version scheme In-Reply-To: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> References: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> Message-ID: Hi Tarek, On Mon, Dec 7, 2009 at 4:53 PM, Tarek Ziad? wrote: > (This is the email I will send to python-dev, so everyone can review > it before it is sent) > > On behalf of the Distutils-SIG, I would like to propose PEP 386 for > inclusion in the sdtlib, and have a final discussion here on > Python-dev. > > http://www.python.org/dev/peps/pep-0386 How does the "Requires-Dist" metadata differ from the current "requires" metadata? The syntax looks the same, why not use "requires"? Should this be explained in the PEP? I think there is a missing bit of information in the PEP: "A class method called from_parts is available if you want to create an instance by providing the parts that composes the version. Each part is a tuple and there are three parts: * the main version part * the pre-release part" What is the third part? Darren From floris.bruynooghe at gmail.com Mon Dec 7 23:17:08 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Mon, 7 Dec 2009 22:17:08 +0000 Subject: [Distutils] RFC822, PKG-INFO and the Description field In-Reply-To: <94bdd2610912061222h49eb0cb4te8887b88d509bb25@mail.gmail.com> References: <94bdd2610912041849m1b140b8fu8a3faf1f6f478ff4@mail.gmail.com> <1260117458.6526.28.camel@localhost> <94bdd2610912061222h49eb0cb4te8887b88d509bb25@mail.gmail.com> Message-ID: <20091207221708.GA27161@laurie.devork> On Sun, Dec 06, 2009 at 09:22:32PM +0100, Tarek Ziad? wrote: > On Sun, Dec 6, 2009 at 5:37 PM, Ronny Pfannschmidt > wrote: > [..] > > > > How about turning that file into a real mime message instead of just a > > set of pseudo mime headers with some pseudo encodings for multiline > > stuff. > > > > That way the long description could be the body of that message, > > no more messy recoding needed. > > > > As far as i can tell, all the other optional fields are designed to fit > > single lines anyway. > > > > The problem is, we may have in the future more multi-line fields so > I think we should not use a message-like body. Could easily hack that in using multipart/alternative messages, given each alternative a meaning. :-) > OTHO we could drop RFC 822 completely and go for a simpler format like > json for 1.2.. That would be a less strange option. If the complete compatibility break does no harm then it might be a good option. But I'm not sure which are all the tools that would be consuming this, it might be risky. Note that stuffing it in multipart/alternative messages suffers the same problem, only your encoding/decoding doesn't. PS: As another (possibly bad) alternative you could encode it in base64. Same compatibility problems however, only your proposal doesn't have those AFAIK. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From floris.bruynooghe at gmail.com Mon Dec 7 23:23:38 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Mon, 7 Dec 2009 22:23:38 +0000 Subject: [Distutils] Proposing PEP 386 - a standard version scheme In-Reply-To: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> References: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> Message-ID: <20091207222338.GB27161@laurie.devork> On Mon, Dec 07, 2009 at 10:53:11PM +0100, Tarek Ziad? wrote: > As I expected, we didn't find a full consensus on the final PEP 386 > schema on Distutils-SIG. But the one presented is > good enough for what we need to express, as far as I am concerned (and > some other people). > > This status understandable, and can go on for months in distutils-SIG. Your brain might have thought of more then your fingers wrote here... Not sure what that's supposed to mean but the last sentence doesn't look complete to me. Otherwise I agree with the content Cheers for the work Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From ziade.tarek at gmail.com Mon Dec 7 23:26:23 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 7 Dec 2009 23:26:23 +0100 Subject: [Distutils] Proposing PEP 386 - a standard version scheme In-Reply-To: References: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> Message-ID: <94bdd2610912071426j5dc9247cv2b68029578a65f71@mail.gmail.com> On Mon, Dec 7, 2009 at 11:16 PM, Darren Dale wrote: > Hi Tarek, > > On Mon, Dec 7, 2009 at 4:53 PM, Tarek Ziad? wrote: >> (This is the email I will send to python-dev, so everyone can review >> it before it is sent) >> >> On behalf of the Distutils-SIG, I would like to propose PEP 386 for >> inclusion in the sdtlib, and have a final discussion here on >> Python-dev. >> >> http://www.python.org/dev/peps/pep-0386 > > How does the "Requires-Dist" metadata differ from the current > "requires" metadata? The syntax looks the same, why not use > "requires"? Should this be explained in the PEP? The "requires" metadata was not referring to distributions at PyPI, but to any module or package. This is explained in PEP 345, but we can add a note in PEP 386, I'll do it > > I think there is a missing bit of information in the PEP: > > "A class method called from_parts is available if you want to create > an instance by providing the parts that composes the version. > > Each part is a tuple and there are three parts: > > * the main version part > * the pre-release part" > > What is the third part? Good catch, a bit is missing (the third part is the dev marker), I'll fix it in the PEP, You can read the API here: http://bitbucket.org/tarek/distutilsversion/src/tip/verlib.py Thanks for the feedback Tarek From ziade.tarek at gmail.com Mon Dec 7 23:42:22 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 7 Dec 2009 23:42:22 +0100 Subject: [Distutils] Proposing PEP 386 - a standard version scheme In-Reply-To: <20091207222338.GB27161@laurie.devork> References: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> <20091207222338.GB27161@laurie.devork> Message-ID: <94bdd2610912071442x71c94a1cn89cf5a3bad46310b@mail.gmail.com> On Mon, Dec 7, 2009 at 11:23 PM, Floris Bruynooghe wrote: > On Mon, Dec 07, 2009 at 10:53:11PM +0100, Tarek Ziad? wrote: >> As I expected, we didn't find a full consensus on the final PEP 386 >> schema on Distutils-SIG. But the one presented is >> good enough for what we need to express, as far as I am concerned (and >> some other people). >> >> This status understandable, and can go on for months in distutils-SIG. > > Your brain might have thought of more then your fingers wrote > here... Not sure what that's supposed to mean but the last sentence > doesn't look complete to me. I'll need some help then on this one, must be my frenglish at work. What I want to say here is that it is understandable that we are not finding a consensus on a version scheme, and that we can bikeshed for ever about it. Bringing the discussion to Python-dev might lead to another round of discussions on the best scheme to adopt, but I eventually expect Guido to end it up with one of these: 1/ We'll pick the "foo" scheme. I accept the pep. 2/ this is not going to work, I reject that pep. (I want 1/ of course!) IOW I think that we already have everything needed to have a useful tool. Tarek From p.f.moore at gmail.com Tue Dec 8 13:50:46 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Dec 2009 12:50:46 +0000 Subject: [Distutils] Proposing PEP 386 - a standard version scheme In-Reply-To: <94bdd2610912071442x71c94a1cn89cf5a3bad46310b@mail.gmail.com> References: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> <20091207222338.GB27161@laurie.devork> <94bdd2610912071442x71c94a1cn89cf5a3bad46310b@mail.gmail.com> Message-ID: <79990c6b0912080450m2ad157adm24e453245a73b776@mail.gmail.com> 2009/12/7 Tarek Ziad? : > On Mon, Dec 7, 2009 at 11:23 PM, Floris Bruynooghe > wrote: >> On Mon, Dec 07, 2009 at 10:53:11PM +0100, Tarek Ziad? wrote: >>> As I expected, we didn't find a full consensus on the final PEP 386 >>> schema on Distutils-SIG. But the one presented is >>> good enough for what we need to express, as far as I am concerned (and >>> some other people). >>> >>> This status understandable, and can go on for months in distutils-SIG. >> >> Your brain might have thought of more then your fingers wrote >> here... Not sure what that's supposed to mean but the last sentence >> doesn't look complete to me. > > I'll need some help then on this one, must be my frenglish at work. "I believe that the current situation is as close to consensus as we will get on distutils-sig, and in the interests of avoiding months of further discussion which won't take things any further, I propose to allow final comments from python-dev and then look for a final decision". How does that sound? Paul. From dsdale24 at gmail.com Tue Dec 8 14:09:04 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 8 Dec 2009 08:09:04 -0500 Subject: [Distutils] Proposing PEP 386 - a standard version scheme In-Reply-To: <79990c6b0912080450m2ad157adm24e453245a73b776@mail.gmail.com> References: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> <20091207222338.GB27161@laurie.devork> <94bdd2610912071442x71c94a1cn89cf5a3bad46310b@mail.gmail.com> <79990c6b0912080450m2ad157adm24e453245a73b776@mail.gmail.com> Message-ID: On Tue, Dec 8, 2009 at 7:50 AM, Paul Moore wrote: > 2009/12/7 Tarek Ziad? : >> On Mon, Dec 7, 2009 at 11:23 PM, Floris Bruynooghe >> wrote: >>> On Mon, Dec 07, 2009 at 10:53:11PM +0100, Tarek Ziad? wrote: >>>> As I expected, we didn't find a full consensus on the final PEP 386 >>>> schema on Distutils-SIG. But the one presented is >>>> good enough for what we need to express, as far as I am concerned (and >>>> some other people). >>>> >>>> This status understandable, and can go on for months in distutils-SIG. >>> >>> Your brain might have thought of more then your fingers wrote >>> here... Not sure what that's supposed to mean but the last sentence >>> doesn't look complete to me. >> >> I'll need some help then on this one, must be my frenglish at work. > > "I believe that the current situation is as close to consensus as we > will get on distutils-sig, and in the interests of avoiding months of > further discussion which won't take things any further, I propose to > allow final comments from python-dev and then look for a final > decision". > > How does that sound? Much better, in my opinion. Darren From ziade.tarek at gmail.com Tue Dec 8 14:41:07 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 8 Dec 2009 14:41:07 +0100 Subject: [Distutils] Proposing PEP 386 - a standard version scheme In-Reply-To: References: <94bdd2610912071353n6d420922n7dba82cf55fee7f2@mail.gmail.com> <20091207222338.GB27161@laurie.devork> <94bdd2610912071442x71c94a1cn89cf5a3bad46310b@mail.gmail.com> <79990c6b0912080450m2ad157adm24e453245a73b776@mail.gmail.com> Message-ID: <94bdd2610912080541y6e811163vcf1c4fae46521fc5@mail.gmail.com> 2009/12/8 Darren Dale : > On Tue, Dec 8, 2009 at 7:50 AM, Paul Moore wrote: >> 2009/12/7 Tarek Ziad? : >>> On Mon, Dec 7, 2009 at 11:23 PM, Floris Bruynooghe >>> wrote: >>>> On Mon, Dec 07, 2009 at 10:53:11PM +0100, Tarek Ziad? wrote: >>>>> As I expected, we didn't find a full consensus on the final PEP 386 >>>>> schema on Distutils-SIG. But the one presented is >>>>> good enough for what we need to express, as far as I am concerned (and >>>>> some other people). >>>>> >>>>> This status understandable, and can go on for months in distutils-SIG. >>>> >>>> Your brain might have thought of more then your fingers wrote >>>> here... Not sure what that's supposed to mean but the last sentence >>>> doesn't look complete to me. >>> >>> I'll need some help then on this one, must be my frenglish at work. >> >> "I believe that the current situation is as close to consensus as we >> will get on distutils-sig, and in the interests of avoiding months of >> further discussion which won't take things any further, I propose to >> allow final comments from python-dev and then look for a final >> decision". >> >> How does that sound? > > Much better, in my opinion. > Thanks Paul ! I'll send it sometimes today Regards Tarek From Rune.Strand at datametrix.no Wed Dec 9 13:32:00 2009 From: Rune.Strand at datametrix.no (Rune Strand) Date: Wed, 9 Dec 2009 13:32:00 +0100 Subject: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe Message-ID: <29649F66EB132748AD74ECF6E32A05C904E7772935@daxdrexch1.dax.local> Hi, setuptools-0.6c11.win32-py2.6.exe will not install on Win 7 with python 2.6.4 installed. Dialog says: "Python version 2.6 is required, which was not found in the registry" I run py32 on win64 C:\>python Python 2.6.4 (r264:75708, Oct 26 2009, 07:36:50) [MSC v.1500 64 bit (AMD64)] on win32 best regards From setuptools at bugs.python.org Wed Dec 9 20:27:06 2009 From: setuptools at bugs.python.org (Daniel Brown) Date: Wed, 09 Dec 2009 19:27:06 +0000 Subject: [Distutils] [issue98] easy_install installs an old version of matplotlib In-Reply-To: <1260386826.88.0.673852673638.issue98@psf.upfronthosting.co.za> Message-ID: <1260386826.88.0.673852673638.issue98@psf.upfronthosting.co.za> New submission from Daniel Brown : 'easy_install matplotlib' results in matplotlib version 0.91.* being installed. The latest version on matplotlib is 0.99.*. I have tried this on Mac and PC (via Cgywin). ---------- messages: 472 nosy: Danielb priority: bug status: unread title: easy_install installs an old version of matplotlib _______________________________________________ Setuptools tracker _______________________________________________ From pje at telecommunity.com Wed Dec 9 22:58:46 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 09 Dec 2009 16:58:46 -0500 Subject: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe In-Reply-To: <29649F66EB132748AD74ECF6E32A05C904E7772935@daxdrexch1.dax. local> References: <29649F66EB132748AD74ECF6E32A05C904E7772935@daxdrexch1.dax.local> Message-ID: <20091209215855.30BDA3A40A8@sparrow.telecommunity.com> At 01:32 PM 12/9/2009 +0100, Rune Strand wrote: >Hi, > >setuptools-0.6c11.win32-py2.6.exe will not install on Win 7 with >python 2.6.4 installed. > >Dialog says: "Python version 2.6 is required, which was not found in >the registry" That seems to be an indication that Python itself is not properly installed on your machine -- it should be in the registry. >I run py32 on win64 > >C:\>python >Python 2.6.4 (r264:75708, Oct 26 2009, 07:36:50) [MSC v.1500 64 bit >(AMD64)] on >win32 How was it installed? Did it come with your PC, or did you install it? From eucci.group at gmail.com Thu Dec 10 00:10:32 2009 From: eucci.group at gmail.com (Jeff Shell) Date: Wed, 9 Dec 2009 16:10:32 -0700 Subject: [Distutils] Problems with setuptools 0.6c11, Mac OS X 10.6, Python 2.4 from MacPorts Message-ID: <88d0d31b0912091510vdb96d5bs9bbe88629cef5394@mail.gmail.com> I've been getting strange behavior with trying to use Buildout with setuptools pinned to 0.6c11 on Mac OS X 10.6, with Python 2.4 built from MacPorts. I'd constantly get the following error: Getting distribution for 'zope.interface==3.5.2'. No eggs found in /var/folders/ta/taVYwGZqHuKHt5V7rcwEzE++-+s/-Tmp-/easy_install-7kLZlO/zope.interface-3.5.2/egg-dist-tmp-dC9GYL (setup script problem?) I first noticed it when trying to use a fresh 'bootstrap.py' from zc.buildout (which I have traditionally not used in our projects), and tried the 'Use Distribute' option. What was interesting was that `zope.interface` 3.5.2 was already installed in my eggs directory, and buildout should not have been trying to get a new one. So this problem seems to show up with both Distribute (0.6.8?) and setuptools (0.6c11). Pinning setuptools to 0.6c9 does not have this problem. I don't know if this is related, but when comparing the setuptools 0.6c9 and 0.6c11 source trees, I noticed a change in how the Mac OS X version is looked up: In setuptools 0.6c11, since the change in `pkg_resources._macosx_vers` to use `platform.mac_ver`, the MacPorts build(s) of Python 2.4 do not seem to be reporting any information out of mac_ver(), causing pkg_resources to report spurious information: >>> import platform >>> import pkg_resources >>> platform.mac_ver() ('', ('', '', ''), '') >>> pkg_resources.get_supported_platform() 'macosx--i386' Notice no Mac OS X version. In setuptools 0.6c9, pkg_resources.get_supported_platform() reports as follows: >>> pkg_resources.get_supported_platform() 'macosx-10.6-i386' (The versions of Python 2.5 and 2.6 supplied by Apple report proper information out of 'platform.mac_ver()', but using those versions causes other problems as versions of 'twisted' start tripping over each other, violently) -- Jeff Shell From hanno at hannosch.eu Thu Dec 10 01:11:28 2009 From: hanno at hannosch.eu (Hanno Schlichting) Date: Thu, 10 Dec 2009 01:11:28 +0100 Subject: [Distutils] Problems with setuptools 0.6c11, Mac OS X 10.6, Python 2.4 from MacPorts In-Reply-To: <88d0d31b0912091510vdb96d5bs9bbe88629cef5394@mail.gmail.com> References: <88d0d31b0912091510vdb96d5bs9bbe88629cef5394@mail.gmail.com> Message-ID: <5cae42b20912091611s189e37feofc8a134062c5dad8@mail.gmail.com> On Thu, Dec 10, 2009 at 12:10 AM, Jeff Shell wrote: > In setuptools 0.6c11, since the change in `pkg_resources._macosx_vers` > to use `platform.mac_ver`, the MacPorts build(s) of Python 2.4 do not > seem to be reporting any information out of mac_ver(), causing > pkg_resources to report spurious information: This has already been reported as a bug in macports at https://trac.macports.org/ticket/22278. Hanno From eucci.group at gmail.com Thu Dec 10 02:19:13 2009 From: eucci.group at gmail.com (Jeff Shell) Date: Wed, 9 Dec 2009 18:19:13 -0700 Subject: [Distutils] Problems with setuptools 0.6c11, Mac OS X 10.6, Python 2.4 from MacPorts In-Reply-To: <5cae42b20912091611s189e37feofc8a134062c5dad8@mail.gmail.com> References: <88d0d31b0912091510vdb96d5bs9bbe88629cef5394@mail.gmail.com> <5cae42b20912091611s189e37feofc8a134062c5dad8@mail.gmail.com> Message-ID: <88d0d31b0912091719p17043957te996c5baa109bb2e@mail.gmail.com> Thanks. That Macports ticket opened me up to the Python buildouts here: http://svn.plone.org/svn/collective/buildout/python/ I'm going to look into that and getting my team using it to build versions of Python we can use easily. MacPorts has been unreliable, lately, in keeping up. As MacPorts is a common tool, it's frustrating (but I guess it's understandable) that setuptools post-0.6c9 breaks on it. On Wed, Dec 9, 2009 at 5:11 PM, Hanno Schlichting wrote: > On Thu, Dec 10, 2009 at 12:10 AM, Jeff Shell wrote: >> In setuptools 0.6c11, since the change in `pkg_resources._macosx_vers` >> to use `platform.mac_ver`, the MacPorts build(s) of Python 2.4 do not >> seem to be reporting any information out of mac_ver(), causing >> pkg_resources to report spurious information: > > This has already been reported as a bug in macports at > https://trac.macports.org/ticket/22278. > > Hanno > -- Jeff Shell From pje at telecommunity.com Thu Dec 10 03:03:00 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 09 Dec 2009 21:03:00 -0500 Subject: [Distutils] Problems with setuptools 0.6c11, Mac OS X 10.6, Python 2.4 from MacPorts In-Reply-To: <88d0d31b0912091510vdb96d5bs9bbe88629cef5394@mail.gmail.com > References: <88d0d31b0912091510vdb96d5bs9bbe88629cef5394@mail.gmail.com> Message-ID: <20091210020310.C9E943A40A8@sparrow.telecommunity.com> At 04:10 PM 12/9/2009 -0700, Jeff Shell wrote: >I've been getting strange behavior with trying to use Buildout with >setuptools pinned to 0.6c11 on Mac OS X 10.6, with Python 2.4 built >from MacPorts. I'd constantly get the following error: > > Getting distribution for 'zope.interface==3.5.2'. > No eggs found in >/var/folders/ta/taVYwGZqHuKHt5V7rcwEzE++-+s/-Tmp-/easy_install-7kLZlO/zope.interface-3.5.2/egg-dist-tmp-dC9GYL >(setup script problem?) > >I first noticed it when trying to use a fresh 'bootstrap.py' from >zc.buildout (which I have traditionally not used in our projects), and >tried the 'Use Distribute' option. What was interesting was that >`zope.interface` 3.5.2 was already installed in my eggs directory, and >buildout should not have been trying to get a new one. > >So this problem seems to show up with both Distribute (0.6.8?) and >setuptools (0.6c11). > >Pinning setuptools to 0.6c9 does not have this problem. > >I don't know if this is related, but when comparing the setuptools >0.6c9 and 0.6c11 source trees, I noticed a change in how the Mac OS X >version is looked up: Yep, that would cause the problem you're showing above. I'll make a note to change the version check code to fall back if mac_ver() gives spurious results. From kiorky at cryptelium.net Thu Dec 10 09:20:21 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 10 Dec 2009 09:20:21 +0100 Subject: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe In-Reply-To: <29649F66EB132748AD74ECF6E32A05C904E7772935@daxdrexch1.dax.local> References: <29649F66EB132748AD74ECF6E32A05C904E7772935@daxdrexch1.dax.local> Message-ID: <4B20AF45.2080908@cryptelium.net> With an explorer go into your python folder Download http://effbot.org/zone/python-register.htm into register.py Put the following code into a file name "register.py" #---------------------------------------------------------------------- # script to register Python 2.0 or later for use with win32all # and other extensions that require Python registry settings # # written by Joakim L?w for Secret Labs AB / PythonWare # # source: # http://www.pythonware.com/products/works/articles/regpy20.htm import sys from _winreg import * # tweak as necessary version = sys.version[:3] installpath = sys.prefix regpath = "SOFTWARE\\Python\\Pythoncore\\%s\\" % (version) installkey = "InstallPath" pythonkey = "PythonPath" pythonpath = "%s;%s\\Lib\\;%s\\DLLs\\" % ( installpath, installpath, installpath ) def RegisterPy(): try: reg = OpenKey(HKEY_LOCAL_MACHINE, regpath) except EnvironmentError: try: reg = CreateKey(HKEY_LOCAL_MACHINE, regpath) SetValue(reg, installkey, REG_SZ, installpath) SetValue(reg, pythonkey, REG_SZ, pythonpath) CloseKey(reg) except: print "*** Unable to register!" return print "--- Python", version, "is now registered!" return if (QueryValue(reg, installkey) == installpath and QueryValue(reg, pythonkey) == pythonpath): CloseKey(reg) print "=== Python", version, "is already registered!" return CloseKey(reg) print "*** Unable to register!" print "*** You probably have another Python installation!" if __name__ == "__main__": RegisterPy() #---------------------------------------------------------------------- Execute the 'Python.exe' file >>> import os >>> os.system('Python register.py') >>> os.system('Python register.py') ... --- Python 2.4 is now registered! Normally your python will be again registered no matter what may have it made not be registered anymore. Rune Strand a ?crit : > Hi, > > setuptools-0.6c11.win32-py2.6.exe will not install on Win 7 with python 2.6.4 installed. > > Dialog says: "Python version 2.6 is required, which was not found in the registry" > > > I run py32 on win64 > > C:\>python > Python 2.6.4 (r264:75708, Oct 26 2009, 07:36:50) [MSC v.1500 64 bit (AMD64)] on > win32 > > best regards > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From ziade.tarek at gmail.com Thu Dec 10 12:55:10 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 10 Dec 2009 12:55:10 +0100 Subject: [Distutils] Problems with setuptools 0.6c11, Mac OS X 10.6, Python 2.4 from MacPorts In-Reply-To: <5cae42b20912091611s189e37feofc8a134062c5dad8@mail.gmail.com> References: <88d0d31b0912091510vdb96d5bs9bbe88629cef5394@mail.gmail.com> <5cae42b20912091611s189e37feofc8a134062c5dad8@mail.gmail.com> Message-ID: <94bdd2610912100355h503bc3f0nbcb17b6ceacf7d27@mail.gmail.com> On Thu, Dec 10, 2009 at 1:11 AM, Hanno Schlichting wrote: > On Thu, Dec 10, 2009 at 12:10 AM, Jeff Shell wrote: >> In setuptools 0.6c11, since the change in `pkg_resources._macosx_vers` >> to use `platform.mac_ver`, the MacPorts build(s) of Python 2.4 do not >> seem to be reporting any information out of mac_ver(), causing >> pkg_resources to report spurious information: > > This has already been reported as a bug in macports at > https://trac.macports.org/ticket/22278. This problem was reported as well in Distribute: http://bitbucket.org/tarek/distribute/issue/92/no-eggs-found I'll add a fallback in case mac_ver returns an empty string From ziade.tarek at gmail.com Thu Dec 10 13:34:19 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 10 Dec 2009 13:34:19 +0100 Subject: [Distutils] Problems with setuptools 0.6c11, Mac OS X 10.6, Python 2.4 from MacPorts In-Reply-To: <88d0d31b0912091719p17043957te996c5baa109bb2e@mail.gmail.com> References: <88d0d31b0912091510vdb96d5bs9bbe88629cef5394@mail.gmail.com> <5cae42b20912091611s189e37feofc8a134062c5dad8@mail.gmail.com> <88d0d31b0912091719p17043957te996c5baa109bb2e@mail.gmail.com> Message-ID: <94bdd2610912100434o69b8d946q3ae5854ed6f36120@mail.gmail.com> On Thu, Dec 10, 2009 at 2:19 AM, Jeff Shell wrote: > Thanks. That Macports ticket opened me up to the Python buildouts here: > > http://svn.plone.org/svn/collective/buildout/python/ > > I'm going to look into that and getting my team using it to build > versions of Python we can use easily. MacPorts has been unreliable, > lately, in keeping up. As MacPorts is a common tool, it's frustrating > (but I guess it's understandable) that setuptools post-0.6c9 breaks on > it. I have fixed it now in Distribute. It will try to read the Mac Version directly if platform.mac_ver() fails as a fallback. http://bitbucket.org/tarek/distribute/issue/92/no-eggs-found The next Distribute version is going to be shipped probably this week-end, so an alternative is to use it instead of building Python, which might be quite painful. Regards Tarek From ziade.tarek at gmail.com Thu Dec 10 13:47:43 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 10 Dec 2009 13:47:43 +0100 Subject: [Distutils] Upcoming Distribute release Message-ID: <94bdd2610912100447g3ad68a81y5443b9b7385047f3@mail.gmail.com> Hey, Quite a few bugs have been fixed lately, and I am planning to fix some more this week end and cut a release. If there's a bug in http://bitbucket.org/tarek/distribute/issues/ you would like to see fixed, speak up ! Regards Tarek From dsdale24 at gmail.com Thu Dec 10 18:18:39 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Thu, 10 Dec 2009 12:18:39 -0500 Subject: [Distutils] question about future pip capabilities Message-ID: Hello, I apologize for cross-posting. I am working on improving how I distribute a few python packages, and in the process I have run into some questions that involve easy_install, setuptools/distribute/, pip, and the relationships between them. I was originally planning on using easy_install to take advantage of extras (easy_install foo[bar]) and automatically install dependencies, but I just discovered that easy_install does not support post-installation routines, which I need in order to add entries to the windows start menu. Pip will, I think, support post_installation routines if I implement a custom install command and pass it to setup(). But pip does not yet support extras or installation from windows binary installers, and does not support installation from eggs. The pip documentation states: * It cannot install from eggs. It only installs from source. (In the future it would be good if it could install binaries from Windows .exe or .msi -- binary install on other platforms is not a priority.) * It doesn't understand Setuptools extras (like package[test]). This should be added eventually. * It is incompatible with some packages that customize distutils or setuptools in their setup.py files. * Maybe it doesn't work on Windows. At least, the author doesn't test on Windows often. Questions concerning pip: * Is there a roadmap or timeline concerning installing from windows exe or msi files? I didn't see anything at bitbucket. * Is there a roadmap or timeline concerning extras? I didn't see anything at bitbucket. * Could anyone please expand on the comment that pip is incompatible with some packages that customize distutils or setuptools in their setup.py files, and that it maybe doesn't work on windows? Questions concerning Distribute: * The documentation states that easy_install will be deprecated, and that we should use pip. Isn't this premature, given the current discrepancy between the capabilities and platform support of pip and easy_install? Questions for pip and Distribute: * How closely coupled are these projects? * I have a bit of time after hours, how can I help? Darren From alex.gronholm at nextday.fi Thu Dec 10 19:23:02 2009 From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=) Date: Thu, 10 Dec 2009 20:23:02 +0200 Subject: [Distutils] question about future pip capabilities In-Reply-To: References: Message-ID: <4B213C86.1060500@nextday.fi> 10.12.2009 19:18, Darren Dale kirjoitti: > Hello, > > I apologize for cross-posting. I am working on improving how I > distribute a few python packages, and in the process I have run into > some questions that involve easy_install, setuptools/distribute/, pip, > and the relationships between them. > > I was originally planning on using easy_install to take advantage of > extras (easy_install foo[bar]) and automatically install dependencies, > but I just discovered that easy_install does not support > post-installation routines, which I need in order to add entries to > the windows start menu. Pip will, I think, support post_installation > routines if I implement a custom install command and pass it to > setup(). But pip does not yet support extras or installation from > windows binary installers, and does not support installation from > eggs. The pip documentation states: > > * It cannot install from eggs. It only installs from source. (In the > future it would be good if it could install binaries from Windows .exe > or .msi -- binary install on other platforms is not a priority.) > * It doesn't understand Setuptools extras (like package[test]). This > should be added eventually. > * It is incompatible with some packages that customize distutils or > setuptools in their setup.py files. > * Maybe it doesn't work on Windows. At least, the author doesn't test > on Windows often. > > Questions concerning pip: > > * Is there a roadmap or timeline concerning installing from windows > exe or msi files? I didn't see anything at bitbucket. > * Is there a roadmap or timeline concerning extras? I didn't see > anything at bitbucket. > * Could anyone please expand on the comment that pip is incompatible > with some packages that customize distutils or setuptools in their > setup.py files, and that it maybe doesn't work on windows? > > Questions concerning Distribute: > > * The documentation states that easy_install will be deprecated, and > that we should use pip. Isn't this premature, given the current > discrepancy between the capabilities and platform support of pip and > easy_install? > The next major distribute release (the one that nixes easy_install) is relatively far in the future, and we (well, I at least) expect pip to have matured enough by then. > Questions for pip and Distribute: > > * How closely coupled are these projects? > * I have a bit of time after hours, how can I help? > > Darren > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From ianb at colorstudy.com Thu Dec 10 20:30:47 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 10 Dec 2009 13:30:47 -0600 Subject: [Distutils] [venv] question about future pip capabilities In-Reply-To: References: Message-ID: On Thu, Dec 10, 2009 at 11:18 AM, Darren Dale wrote: > Questions concerning pip: > > * Is there a roadmap or timeline concerning installing from windows > exe or msi files? I didn't see anything at bitbucket. There isn't a timeline, as we don't have anyone actively contributing with respect to Windows. > * Is there a roadmap or timeline concerning extras? I didn't see > anything at bitbucket. There isn't really; extras would be nice, but there hasn't been much demand for them. pip requirement files serve a similar purpose though in a somewhat different way. > * Could anyone please expand on the comment that pip is incompatible > with some packages that customize distutils or setuptools in their > setup.py files, and that it maybe doesn't work on windows? Maybe the Windows statement is exaggerated. Unfortunately we don't have a good Windows testing setup, and I don't even know what that would look like. Some buildbot (or buildbot-like-thing, e.g., Hudson) running on a Windows machine would be really helpful. pip itself relies only on the command line and some expected behavior of setup.py files, it doesn't really care how this is implemented. To install a package it does: python -c 'import setuptools; __file__="setup.py"; execfile(__file__)' install --single-version-externally-managed --record=/tmp/XXX The first part installs the setuptools monkeypatching for packages that just use distutils. So you at least have to make something compatible with that monkeypatching (i.e., it can't rely on setuptools not being imported). Then it uses those two command-line switches (and I'm pretty sure that's it; you can see a complete record of what pip does with pip install -vv). Those are provided by Setuptools (or Distribute) and not by Distutils, but pip doesn't care how they are implemented so long as you accept those options. Also to be compatible with pip you have to create a proper Package.egg-info/ directory (to record that the package is installed) and use that --record option to write a list of all the files that are written to disk, so they can be removed when the package is removed. > Questions for pip and Distribute: > > * How closely coupled are these projects? They are not closely coupled at all. > * I have a bit of time after hours, how can I help? If you are a Windows developer, Windows help for pip would be great. Installing Windows binaries would be the biggest help; I suspect this means unzipping the .exe or .msi and moving the files into place manually (since there is no setup.py), and maybe even generating that .egg-info/ file. I suspect that if you install a Windows installer directly that pip (and Setuptools/Distribute/pkg_resources) might not be able to see that the package is installed. But I've never tried. It would be nice if these installation techniques were compatible with each other, which means the installer creating .egg-info files (or shipping with EGG-INFO, similar to how Windows binary eggs work). In theory Windows binary eggs could be installed similarly, but I don't think there's as much enthusiasm for that among Windows users, and .exe or .msi packages should be mostly equivalent (except that lack of metadata). But I don't consider myself a good judge of Window users' preferences, so maybe I'm wrong. A shorter time commitment is to download pip and run the tests on Windows and report problems, as it's unfortunately quite possible we've introduced Windows regressions. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker From carl at meyerloewen.net Thu Dec 10 20:40:43 2009 From: carl at meyerloewen.net (Carl Meyer) Date: Thu, 10 Dec 2009 14:40:43 -0500 Subject: [Distutils] [venv] question about future pip capabilities In-Reply-To: References: Message-ID: <4B214EBB.9080705@meyerloewen.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ian Bicking wrote: > A shorter time commitment is to download pip and run the tests on > Windows and report problems, as it's unfortunately quite possible > we've introduced Windows regressions. I did a bit of work on getting pip tests running under Windows last spring, but didn't get very far. There were many test failures, most of them seemingly superficial: posix-specific assumptions about file locations and path separators in the tests themselves, not necessarily broken functionality in pip (though it's hard to say for sure with broken tests). Fixing those issues isn't hard, just tedious. I don't know what deeper issues or regressions might be uncovered once the tests run. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFLIU67FRcxmeyPUXIRAlt5AJoCVlEMF8ZucNPPwypKWETDhe98ZQCfeT5P fs8t8Mr0jaj2gsEwNN3kyvo= =fgc1 -----END PGP SIGNATURE----- From ianb at colorstudy.com Thu Dec 10 22:09:29 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 10 Dec 2009 15:09:29 -0600 Subject: [Distutils] [venv] question about future pip capabilities In-Reply-To: <4B214EBB.9080705@meyerloewen.net> References: <4B214EBB.9080705@meyerloewen.net> Message-ID: On Thu, Dec 10, 2009 at 1:40 PM, Carl Meyer wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Ian Bicking wrote: >> A shorter time commitment is to download pip and run the tests on >> Windows and report problems, as it's unfortunately quite possible >> we've introduced Windows regressions. > > I did a bit of work on getting pip tests running under Windows last > spring, but didn't get very far. There were many test failures, most of > them seemingly superficial: posix-specific assumptions about file > locations and path separators in the tests themselves, not necessarily > broken functionality in pip (though it's hard to say for sure with > broken tests). Fixing those issues isn't hard, just tedious. I don't > know what deeper issues or regressions might be uncovered once the tests > run. Probably some of those issues could also be fixed in ScriptTest, instead of making the pip tests tediously platform abstracted (e.g., normalizing \ vs /). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker From Rune.Strand at datametrix.no Thu Dec 10 22:23:48 2009 From: Rune.Strand at datametrix.no (Rune Strand) Date: Thu, 10 Dec 2009 22:23:48 +0100 Subject: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe In-Reply-To: <20091209215855.30BDA3A40A8@sparrow.telecommunity.com> References: <29649F66EB132748AD74ECF6E32A05C904E7772935@daxdrexch1.dax.local> <20091209215855.30BDA3A40A8@sparrow.telecommunity.com> Message-ID: <29649F66EB132748AD74ECF6E32A05C904E7772B48@daxdrexch1.dax.local> Subject: Re: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe At 01:32 PM 12/9/2009 +0100, Rune Strand wrote: >>Hi, >> >>setuptools-0.6c11.win32-py2.6.exe will not install on Win 7 with >>python 2.6.4 installed. >> >>Dialog says: "Python version 2.6 is required, which was not found in >>the registry" >That seems to be an indication that Python itself is not properly >installed on your machine -- it should be in the registry. Python exists in HKLM/Software. However, all keys but HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.6\InstallPath and HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.6\PythonPath are empty. >>I run py32 on win64 >> >>C:\>python >>Python 2.6.4 (r264:75708, Oct 26 2009, 07:36:50) [MSC v.1500 64 bit >>(AMD64)] on >>win32 >How was it installed? Did it come with your PC, or did you install it? I installed it using the .MSI from python.org. If this is some anomaly with my installation, I will simply uninstall etc. Thanks From Rune.Strand at datametrix.no Thu Dec 10 22:28:56 2009 From: Rune.Strand at datametrix.no (Rune Strand) Date: Thu, 10 Dec 2009 22:28:56 +0100 Subject: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe In-Reply-To: <4B20AF45.2080908@cryptelium.net> References: <29649F66EB132748AD74ECF6E32A05C904E7772935@daxdrexch1.dax.local> <4B20AF45.2080908@cryptelium.net> Message-ID: <29649F66EB132748AD74ECF6E32A05C904E7772B4A@daxdrexch1.dax.local> >Subject: Re: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe >With an explorer go into your python folder Download http://effbot.org/zone/python-register.htm into register.py <-- snip >Put the following code into a file name "register.py" >#---------------------------------------------------------------------- ># script to register Python 2.0 or later for use with win32all ... >Normally your python will be again registered no matter what may have it made not be registered anymore. <-- snip Thanks, but I'm afraid it didn't do the trick. Python already exist in the registry. I compared with other installation, and it seems identical. Output from the script you posted: >>> os.system("Python register.py") *** Unable to register! *** You probably have another Python installation! 0 br From dsdale24 at gmail.com Thu Dec 10 22:58:48 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Thu, 10 Dec 2009 16:58:48 -0500 Subject: [Distutils] [venv] question about future pip capabilities In-Reply-To: References: Message-ID: On Thu, Dec 10, 2009 at 2:30 PM, Ian Bicking wrote: > On Thu, Dec 10, 2009 at 11:18 AM, Darren Dale wrote: >> Questions concerning pip: >> >> * Is there a roadmap or timeline concerning installing from windows >> exe or msi files? I didn't see anything at bitbucket. > > There isn't a timeline, as we don't have anyone actively contributing > with respect to Windows. > >> * Is there a roadmap or timeline concerning extras? I didn't see >> anything at bitbucket. > > There isn't really; extras would be nice, but there hasn't been much > demand for them. ?pip requirement files serve a similar purpose though > in a somewhat different way. I brought this up on the virtualenv mailing list once before. Requirements files serve a very different use-case. If I want to define a project extra like foo[traits_qt4] that depends on a project with an extra like traits[qt4] > 3.2.0, I can't just list traits[qt4] > 3.2.0 in my requirements file, because pip is not designed to recursively check the (arbitrarily-named) requirements files of the dependencies I declare in my own requirements file. I have to keep track of not only my projects dependencies, but my dependencies dependencies, and on and on. >> * I have a bit of time after hours, how can I help? > > If you are a Windows developer, Windows help for pip would be great. I distribute windows installers for a couple projects, but I'm not a windows developer... > A shorter time commitment is to download pip and run the tests on > Windows and report problems, as it's unfortunately quite possible > we've introduced Windows regressions. This sounds like a good place to start. Regards, Darren From dsdale24 at gmail.com Thu Dec 10 23:03:48 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Thu, 10 Dec 2009 17:03:48 -0500 Subject: [Distutils] question about future pip capabilities In-Reply-To: <4B213C86.1060500@nextday.fi> References: <4B213C86.1060500@nextday.fi> Message-ID: 2009/12/10 Alex Gr?nholm : > 10.12.2009 19:18, Darren Dale kirjoitti: >> * The documentation states that easy_install will be deprecated, and >> that we should use pip. Isn't this premature, given the current >> discrepancy between the capabilities and platform support of pip and >> easy_install? >> > > The next major distribute release (the one that nixes easy_install) is > relatively far in the future, and we (well, I at least) expect pip to have > matured enough by then. What gives you that confidence? Ian Bicking just responded that there is currently not a timeline for full windows support, installing from binaries on windows, or project extras. Darren From ziade.tarek at gmail.com Thu Dec 10 23:12:00 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 10 Dec 2009 23:12:00 +0100 Subject: [Distutils] question about future pip capabilities In-Reply-To: References: Message-ID: <94bdd2610912101412o3ae6682cj41299379ba1dcb84@mail.gmail.com> 2009/12/10 Darren Dale : [..] > Questions for pip and Distribute: > > * How closely coupled are these projects? As Ian said, they are not closely coupled, but some people are contributing to both projects, and in Distribute we want to stop developing easy_install because a package installer should be project on its own. And pip development is very active these days. IOW if Pip needs apis Distutils doesn't (yet) provide, Distribute will try to provide them, > * I have a bit of time after hours, how can I help? If you want to help on Distribute, there are many things you could do, from bug fixes to new development in the 0.7 versions. Contact me online if you want to discuss this (#distutils on freenode) Tarek From ziade.tarek at gmail.com Thu Dec 10 23:28:29 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 10 Dec 2009 23:28:29 +0100 Subject: [Distutils] question about future pip capabilities In-Reply-To: References: <4B213C86.1060500@nextday.fi> Message-ID: <94bdd2610912101428n39f2700coa1452b147b28ef7b@mail.gmail.com> 2009/12/10 Darren Dale : > 2009/12/10 Alex Gr?nholm : >> 10.12.2009 19:18, Darren Dale kirjoitti: >>> * The documentation states that easy_install will be deprecated, and >>> that we should use pip. Isn't this premature, given the current >>> discrepancy between the capabilities and platform support of pip and >>> easy_install? >>> >> >> The next major distribute release (the one that nixes easy_install) is >> relatively far in the future, and we (well, I at least) expect pip to have >> matured enough by then. > > What gives you that confidence? Ian Bicking just responded that there > is currently not a timeline for full windows support, installing from > binaries on windows, or project extras. He said that because no one worked on it yet. By the time we will want to release Distribute 0.7 (without easy_install), if pip doesn't have what is required to support win32, I (or some other Distribute dev) will try to contribute to pip to help making it happen (if no one did before). IOW, the release of Distribute 0.7.x will be done once pip has what is required todrop easy_install for good Tarek From myersjj at gmail.com Fri Dec 11 23:47:37 2009 From: myersjj at gmail.com (James J Myers) Date: Fri, 11 Dec 2009 14:47:37 -0800 Subject: [Distutils] Problem with buildout Message-ID: <4B22CC09.3050307@pacbell.net> I've been using bootstrap.py/buildout happily for a while now, but recently (early December) my buildout stopped working. The symptom is that it appears to fail to unpack Apache as shown in output below. I'm not aware of changing anything in my scripts that could account for this and I really need this to work! Any ideas gratefully accepted! The recipe segment that is failing is at the bottom of this email... I've tried to override the version of zc.buildout used by bootstrap.py but it seems to always use 1.4.3 anyway (which I notice is new as of Dec 10). Generated interpreter '/home/dhat/webapps/portal_test2/myproject/bin/python'. Installing apache. apache: Downloading http://apache.multihomed.net/httpd/httpd-2.2.14.tar.gz apache: Unpacking and configuring sh: ./configure: /bin/sh: bad interpreter: Permission denied While: Installing apache. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/home/dhat/webapps/portal_test2/myproject/eggs/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", line 1660, in main getattr(buildout, command)(args) File "/home/dhat/webapps/portal_test2/myproject/eggs/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", line 532, in install installed_files = self[part]._call(recipe.install) File "/home/dhat/webapps/portal_test2/myproject/eggs/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", line 1204, in _call return f() File "build/bdist.linux-i686/egg/zc/recipe/cmmi/__init__.py", line 159, in install File "build/bdist.linux-i686/egg/zc/recipe/cmmi/__init__.py", line 186, in cmmi File "build/bdist.linux-i686/egg/zc/recipe/cmmi/__init__.py", line 32, in system SystemError: ('Failed', './configure --prefix=/home/dhat/webapps/portal_test2/myproject/parts/apache --enable-proxy=shared --enable-rewrite=shared --enable-auth_digest=shared --enable-alias=shared') Recipe: [apache] recipe = zc.recipe.cmmi #url = http://www.ibiblio.org/pub/mirrors/apache/httpd/httpd-2.2.14.tar.gz url = http://apache.multihomed.net/httpd/httpd-2.2.14.tar.gz extra_options = --enable-proxy=shared --enable-rewrite=shared --enable-auth_digest=shared --enable-alias=shared -------------- next part -------------- A non-text attachment was scrubbed... Name: myersjj.vcf Type: text/x-vcard Size: 270 bytes Desc: not available URL: From encolpe.degoute at free.fr Fri Dec 11 23:59:34 2009 From: encolpe.degoute at free.fr (Encolpe Degoute) Date: Fri, 11 Dec 2009 23:59:34 +0100 Subject: [Distutils] Problem with buildout In-Reply-To: <4B22CC09.3050307@pacbell.net> References: <4B22CC09.3050307@pacbell.net> Message-ID: Hello, Check the zc.recipe.cmmi version. Recent version introduce an incompatibility for recipe using it: http://encolpe.wordpress.com/2009/06/03/heads-up-new-plone-recipe-pound-release/ Regards, James J Myers a ?crit : > I've been using bootstrap.py/buildout happily for a while now, but > recently (early December) my buildout stopped working. > The symptom is that it appears to fail to unpack Apache as shown in > output below. > > I'm not aware of changing anything in my scripts that could account for > this and I really need this to work! > Any ideas gratefully accepted! > > The recipe segment that is failing is at the bottom of this email... > > I've tried to override the version of zc.buildout used by bootstrap.py > but it seems to always use 1.4.3 anyway (which I notice is new as of Dec > 10). > > Generated interpreter > '/home/dhat/webapps/portal_test2/myproject/bin/python'. > Installing apache. > apache: Downloading http://apache.multihomed.net/httpd/httpd-2.2.14.tar.gz > apache: Unpacking and configuring > sh: ./configure: /bin/sh: bad interpreter: Permission denied > While: > Installing apache. > > An internal error occured due to a bug in either zc.buildout or in a > recipe being used: > Traceback (most recent call last): > File > "/home/dhat/webapps/portal_test2/myproject/eggs/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", > line 1660, in main > getattr(buildout, command)(args) > File > "/home/dhat/webapps/portal_test2/myproject/eggs/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", > line 532, in install > installed_files = self[part]._call(recipe.install) > File > "/home/dhat/webapps/portal_test2/myproject/eggs/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", > line 1204, in _call > return f() > File "build/bdist.linux-i686/egg/zc/recipe/cmmi/__init__.py", line > 159, in install > File "build/bdist.linux-i686/egg/zc/recipe/cmmi/__init__.py", line > 186, in cmmi > File "build/bdist.linux-i686/egg/zc/recipe/cmmi/__init__.py", line 32, > in system > SystemError: ('Failed', './configure > --prefix=/home/dhat/webapps/portal_test2/myproject/parts/apache > --enable-proxy=shared --enable-rewrite=shared > --enable-auth_digest=shared --enable-alias=shared') > > > Recipe: > > [apache] > recipe = zc.recipe.cmmi > #url = http://www.ibiblio.org/pub/mirrors/apache/httpd/httpd-2.2.14.tar.gz > url = http://apache.multihomed.net/httpd/httpd-2.2.14.tar.gz > extra_options = > --enable-proxy=shared > --enable-rewrite=shared > --enable-auth_digest=shared > --enable-alias=shared > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Encolpe DEGOUTE http://encolpe.degoute.free.fr/ Logiciels libres, hockey sur glace et autres activit?s c?r?brales From myersjj at gmail.com Sat Dec 12 02:33:48 2009 From: myersjj at gmail.com (James J Myers) Date: Fri, 11 Dec 2009 17:33:48 -0800 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi Message-ID: <4B22F2FC.3070700@pacbell.net> I changed my buildout to use earlier version of zc.recipe.cmmi, back to 1.1.0 and it still fails with the same error. According to your report, 1.1.6 or earlier should work, but not for me. From myersjj at gmail.com Sat Dec 12 03:01:40 2009 From: myersjj at gmail.com (James J Myers) Date: Fri, 11 Dec 2009 18:01:40 -0800 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi Message-ID: <4B22F984.50107@pacbell.net> Next try was to change buildout to use minitage.recipe.cmmi and that works! So it appears that something changed in zc.recipe.cmmi in December that broke it. From tseaver at palladion.com Sat Dec 12 06:14:40 2009 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 12 Dec 2009 00:14:40 -0500 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <4B22F984.50107@pacbell.net> References: <4B22F984.50107@pacbell.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 James J Myers wrote: > Next try was to change buildout to use minitage.recipe.cmmi and that works! > > So it appears that something changed in zc.recipe.cmmi in December that > broke it. Unless you are using an SVN checkout of zc.recipe.cmmi, the latest release was 1.3.1, 2009-09-10. zc.buildout 1.4.3 was released 2009-12-10, but its deltas don't appear to explain the behavior you report. $ export ZSVN=svn+ssh://svn.zope.org/repos/main/ $ svn diff $ZSVN/zc.buildout/tags/1.4.{2,3} Can you reproduce the problem in a "fresh" buildout environment? E.g., check you your buildout, run bootstrap, then buildout? How about if you pin zc.buildout to 1.4.2? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksjJrUACgkQ+gerLs4ltQ7GaQCcCvz7aF6pkxAVseKgmmlszkkm z5IAoMxrJcCw7XmI5yXyt5nXM6bGv098 =BVDW -----END PGP SIGNATURE----- From kiorky at cryptelium.net Sat Dec 12 12:32:31 2009 From: kiorky at cryptelium.net (kiorky) Date: Sat, 12 Dec 2009 12:32:31 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <4B22F984.50107@pacbell.net> References: <4B22F984.50107@pacbell.net> Message-ID: <4B237F4F.2090703@cryptelium.net> James J Myers a ?crit : > Next try was to change buildout to use minitage.recipe.cmmi and that works! Great, you can also check for all the other features this recipe provides :). (And also for minitage stuff around, specially if you are building 'non python stuff' in prefix, it can help you to set your compilation flags and things like rpath.) > > So it appears that something changed in zc.recipe.cmmi in December that > broke it. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From ziade.tarek at gmail.com Sat Dec 12 13:33:59 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 13:33:59 +0100 Subject: [Distutils] Distribute 0.6.9 released Message-ID: <94bdd2610912120433q1fd04e7fi1fc017aa923daee2@mail.gmail.com> Hi, On behalf of the Distribute team, I am happy to release Distribute 0.6.9. Important changes: * Distribute now works fine under some versions of MacPort Python that have a bug with the platform.mac_ver() API (see http://trac.macports.org/ticket/22278) * Fixed easy_install (the command) behavior w.r.t. a local setup.cfg * Now Sandbox can allow writing in some specific files (by default : os.devnull) Detailed Changes: * Issue 90: unknown setuptools version can be added in the working set * Issue 87: setupt.py doesn't try to convert distribute_setup.py anymore Initial Patch by arfrever. * Issue 89: added a side bar with a download link to the doc. * Issue 86: fixed missing sentence in pkg_resources doc. * Added a nicer error message when a DistributionNotFound is raised. * Issue 80: test_develop now works with Python 3.1 * Issue 93: upload_docs now works if there is an empty sub-directory. * Issue 70: exec bit on non-exec files * Issue 99: now the standalone easy_install command doesn't uses a "setup.cfg" if any exists in the working directory. It will use it only if triggered by ``install_requires`` from a setup.py call (install, develop, etc). * Issue 101: Allowing ``os.devnull`` in Sandbox * Issue 92: Fixed the "no eggs" found error with MacPort (platform.mac_ver() fails) * Issue 103: test_get_script_header_jython_workaround not run anymore under py3 with C or POSIX local. Contributed by Arfrever. * Issue 104: remvoved the assertion when the installation fails, with a nicer message for the end user. * Issue 100: making sure there's no SandboxViolation when the setup script patches setuptools. Regards, Tarek -- Tarek Ziad? | http://ziade.org From reinout at vanrees.org Sat Dec 12 14:56:55 2009 From: reinout at vanrees.org (Reinout van Rees) Date: Sat, 12 Dec 2009 14:56:55 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: References: <4B22F984.50107@pacbell.net> Message-ID: On 12/12/09 6:14 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > James J Myers wrote: >> Next try was to change buildout to use minitage.recipe.cmmi and that works! >> >> So it appears that something changed in zc.recipe.cmmi in December that >> broke it. > > Unless you are using an SVN checkout of zc.recipe.cmmi, the latest > release was 1.3.1, 2009-09-10. zc.buildout 1.4.3 was released > 2009-12-10, but its deltas don't appear to explain the behavior you report. I'm also getting an error: While: Installing. Getting section lxml. Initializing section lxml. Installing recipe z3c.recipe.staticlxml >= 0.6. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", line 1660, in main File "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", line 416, in install File "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", line 964, in __getitem__ File "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", line 1048, in _initialize File "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", line 1004, in _install_and_load File "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", line 800, in install File "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", line 663, in install File "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", line 612, in _constrain AttributeError: 'str' object has no attribute 'project_name' I put a try/except in there and 'str' in this case is the string "The 'zc.recipe.cmmi' distribution was not found on this system, and is required by this application.". The staticlxml recipe uses zc.recipe.cmmi. The path to zc.buildout that's in the traceback isn't correct either: it is simply a shared egg, definitively not something in a temp folder :-) Weird... Well, I'm off cycling. I'll dig around a bit in zc.recipe.cmmi to see if it perhaps uses the basically one line in zc.buildout that changed with 1.4.3. Another option: does it have to do with distribute 0.6.9 that was just released? I pinned zc.buildout to 1.4.3 to allow it to update distribute from 0.6.8 to 0.6.9 (that was what the buildout fix was for, otherwise you'd get an infinite recursion). Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets" From ziade.tarek at gmail.com Sat Dec 12 15:06:13 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 15:06:13 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: References: <4B22F984.50107@pacbell.net> Message-ID: <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> On Sat, Dec 12, 2009 at 2:56 PM, Reinout van Rees wrote: > On 12/12/09 6:14 AM, Tres Seaver wrote: >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> James J Myers wrote: >>> >>> Next try was to change buildout to use minitage.recipe.cmmi and that >>> works! >>> >>> So it appears that something changed in zc.recipe.cmmi in December that >>> broke it. >> >> Unless you are using an SVN checkout of zc.recipe.cmmi, the latest >> release was 1.3.1, 2009-09-10. ?zc.buildout 1.4.3 was released >> 2009-12-10, but its deltas don't appear to explain the behavior you >> report. > > I'm also getting an error: > > While: > ?Installing. > ?Getting section lxml. > ?Initializing section lxml. > ?Installing recipe z3c.recipe.staticlxml >= 0.6. > > An internal error occured due to a bug in either zc.buildout or in a > recipe being used: > Traceback (most recent call last): > ?File > "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", > line 1660, in main > ?File > "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", > line 416, in install > ?File > "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", > line 964, in __getitem__ > ?File > "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", > line 1048, in _initialize > ?File > "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", > line 1004, in _install_and_load > ?File > "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", > line 800, in install > ?File > "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", > line 663, in install > ?File > "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", > line 612, in _constrain > AttributeError: 'str' object has no attribute 'project_name' > > > I put a try/except in there and 'str' in this case is the string "The > 'zc.recipe.cmmi' distribution was not found on this system, and is required > by this application.". > > The staticlxml recipe uses zc.recipe.cmmi. > > > The path to zc.buildout that's in the traceback isn't correct either: it is > simply a shared egg, definitively not something in a temp folder :-) > > > Weird... Well, I'm off cycling. I'll dig around a bit in zc.recipe.cmmi to > see if it perhaps uses the basically one line in zc.buildout that changed > with 1.4.3. > > Another option: does it have to do with distribute 0.6.9 that was just > released? I pinned zc.buildout to 1.4.3 to allow it to update distribute > from 0.6.8 to 0.6.9 (that was what the buildout fix was for, otherwise you'd > get an infinite recursion). > Yes that's a problem introduced by 0.6.9. I have changed the way the DistributionNotFound() exception is raised. Unfortunately zc.buildout does a str(error) to grab the name of the distribution, leading to the problem you have I am reverting this change and releasing 0.6.10 so people are not confused. Then we can think about changing zc.buildout error handling. Cheers, From ziade.tarek at gmail.com Sat Dec 12 15:25:39 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 15:25:39 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> Message-ID: <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> On Sat, Dec 12, 2009 at 3:06 PM, Tarek Ziad? wrote: > On Sat, Dec 12, 2009 at 2:56 PM, Reinout van Rees wrote: >> On 12/12/09 6:14 AM, Tres Seaver wrote: >>> >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA1 >>> >>> James J Myers wrote: >>>> >>>> Next try was to change buildout to use minitage.recipe.cmmi and that >>>> works! >>>> >>>> So it appears that something changed in zc.recipe.cmmi in December that >>>> broke it. >>> >>> Unless you are using an SVN checkout of zc.recipe.cmmi, the latest >>> release was 1.3.1, 2009-09-10. ?zc.buildout 1.4.3 was released >>> 2009-12-10, but its deltas don't appear to explain the behavior you >>> report. >> >> I'm also getting an error: >> >> While: >> ?Installing. >> ?Getting section lxml. >> ?Initializing section lxml. >> ?Installing recipe z3c.recipe.staticlxml >= 0.6. >> >> An internal error occured due to a bug in either zc.buildout or in a >> recipe being used: >> Traceback (most recent call last): >> ?File >> "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", >> line 1660, in main >> ?File >> "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", >> line 416, in install >> ?File >> "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", >> line 964, in __getitem__ >> ?File >> "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", >> line 1048, in _initialize >> ?File >> "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/buildout.py", >> line 1004, in _install_and_load >> ?File >> "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", >> line 800, in install >> ?File >> "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", >> line 663, in install >> ?File >> "/private/var/folders/qC/qC6d69l0EDe-sx-yyKchqU+++TI/-Tmp-/tmpeuNbOC/zc.buildout-1.4.3-py2.5.egg/zc/buildout/easy_install.py", >> line 612, in _constrain >> AttributeError: 'str' object has no attribute 'project_name' Ok this is fixed now. 0.6.10 released I think we should work on adding a functional test in Distribute, that runs some zc.buildout scripts to make sure this doesn't happen again, Reinout, do you have some buildout.cfg scripts you could add in Distribute, that can be built quickly and illustrates this problem ? Tarek From yonsy at aureal.com.pe Sat Dec 12 16:45:05 2009 From: yonsy at aureal.com.pe (Yonsy Solis) Date: Sat, 12 Dec 2009 10:45:05 -0500 Subject: [Distutils] problem with buildout & distribute Message-ID: <1260632705.7192.6.camel@hades.blackhandchronicles.homeip.net> I have problem im a new machine with a pristine python install i download the bootstrap.py to apply to a recipe that i use in other office, when i try to run with the --distribute option i have this error. Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.11.tar.gz Traceback (most recent call last): File "bootstrap.py", line 67, in ez['use_setuptools'](to_dir=tmpeggs, download_delay=0, no_fake=True) ... ... File "/usr/lib/python2.5/urllib2.py", line 506, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 404: Not Found the problem is in the distribute_setup.py file in http://python-distribute.org/distribute_setup.py in line 49 DEFAULT_VERSION = "0.6.11" define a version of distribute egg that dont exist (yet?) -- Yonsy Solis Aureal Systems (mov) 989-124-141 (www) http://www.aureal.com.pe From ziade.tarek at gmail.com Sat Dec 12 17:06:43 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 17:06:43 +0100 Subject: [Distutils] problem with buildout & distribute In-Reply-To: <1260632705.7192.6.camel@hades.blackhandchronicles.homeip.net> References: <1260632705.7192.6.camel@hades.blackhandchronicles.homeip.net> Message-ID: <94bdd2610912120806l676ebbaer3d0d9f35adeec2f9@mail.gmail.com> Hi I uploaded the wrong script. I have fixed this. Thanks for the feedback On Dec 12, 2009 5:03 PM, "Yonsy Solis" wrote: I have problem im a new machine with a pristine python install i download the bootstrap.py to apply to a recipe that i use in other office, when i try to run with the --distribute option i have this error. Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.11.tar.gz Traceback (most recent call last): File "bootstrap.py", line 67, in ez['use_setuptools'](to_dir=tmpeggs, download_delay=0, no_fake=True) ... ... File "/usr/lib/python2.5/urllib2.py", line 506, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 404: Not Found the problem is in the distribute_setup.py file in http://python-distribute.org/distribute_setup.py in line 49 DEFAULT_VERSION = "0.6.11" define a version of distribute egg that dont exist (yet?) -- Yonsy Solis Aureal Systems (mov) 989-124-141 (www) http://www.aureal.com.pe _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From kiorky at cryptelium.net Sat Dec 12 17:08:25 2009 From: kiorky at cryptelium.net (kiorky) Date: Sat, 12 Dec 2009 17:08:25 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> Message-ID: <4B23BFF9.90705@cryptelium.net> Tarek Ziad? a ?crit : > On Sat, Dec 12, 2009 at 3:06 PM, Tarek Ziad? wrote: >> On Sat, Dec 12, 2009 at 2:56 PM, Reinout van Rees wrote: >> > Ok this is fixed now. 0.6.10 released > > I think we should work on adding a functional test in Distribute, that > runs some zc.buildout scripts > to make sure this doesn't happen again This can break zc.recipe.egg also, other relative recipes around, and other stuff doing magic with 'setuptools/pkg_resources'. Is there no other way to break backward compatibility, in the 0.6.x? > > Reinout, do you have some buildout.cfg scripts you could add in > Distribute, that can be built > quickly and illustrates this problem ? > > Tarek > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From reinout at vanrees.org Sat Dec 12 17:16:55 2009 From: reinout at vanrees.org (Reinout van Rees) Date: Sat, 12 Dec 2009 17:16:55 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> Message-ID: On 12/12/09 3:06 PM, Tarek Ziad? wrote: > On Sat, Dec 12, 2009 at 2:56 PM, Reinout van Rees wrote: >> AttributeError: 'str' object has no attribute 'project_name' >> >> I put a try/except in there and 'str' in this case is the string "The >> 'zc.recipe.cmmi' distribution was not found on this system, and is required >> by this application.". > > Yes that's a problem introduced by 0.6.9. > > I have changed the way the DistributionNotFound() exception is raised. > Unfortunately > zc.buildout does a str(error) to grab the name of the distribution, > leading to the problem you have > > I am reverting this change and releasing 0.6.10 so people are not confused. Thanks, my buildout runs fine now. And I'm happy to see that zc.buildout 1.4.3 indeed allows a distribute upgrade just fine without infinite recursion :-) Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets" From ziade.tarek at gmail.com Sat Dec 12 17:26:28 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 17:26:28 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <4B23BFF9.90705@cryptelium.net> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> <4B23BFF9.90705@cryptelium.net> Message-ID: <94bdd2610912120826s6a016977re3fc583424c30ff1@mail.gmail.com> On Sat, Dec 12, 2009 at 5:08 PM, kiorky wrote: > > > Tarek Ziad? a ?crit : >> On Sat, Dec 12, 2009 at 3:06 PM, Tarek Ziad? wrote: >>> On Sat, Dec 12, 2009 at 2:56 PM, Reinout van Rees wrote: >>> >> Ok this is fixed now. 0.6.10 released >> >> I think we should work on adding a functional test in Distribute, that >> runs some zc.buildout scripts >> to make sure this doesn't happen again > > This can break zc.recipe.egg also, other relative recipes around, and other > stuff doing magic with 'setuptools/pkg_resources'. Is there no other way to > break backward compatibility, in the 0.6.x? > There are hundreds of way to break things :) IOW everytime a change is made, there's a potential risk. Functional tests may help Tarek From ssteinerx at gmail.com Sat Dec 12 18:29:59 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sat, 12 Dec 2009 12:29:59 -0500 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <94bdd2610912120826s6a016977re3fc583424c30ff1@mail.gmail.com> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> <4B23BFF9.90705@cryptelium.net> <94bdd2610912120826s6a016977re3fc583424c30ff1@mail.gmail.com> Message-ID: On Dec 12, 2009, at 11:26 AM, Tarek Ziad? wrote: > There are hundreds of way to break things :) > > IOW everytime a change is made, there's a potential risk. Functional > tests may help Tarek, What are you using for a test suite pre-release now? I started on the multi-package installer we talked about but we never got back together to get it into the release testing. Steve From reinout at vanrees.org Sat Dec 12 18:33:58 2009 From: reinout at vanrees.org (Reinout van Rees) Date: Sat, 12 Dec 2009 18:33:58 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> Message-ID: On 12/12/09 3:25 PM, Tarek Ziad? wrote: >>> AttributeError: 'str' object has no attribute 'project_name' > > Ok this is fixed now. 0.6.10 released > > I think we should work on adding a functional test in Distribute, that > runs some zc.buildout scripts > to make sure this doesn't happen again, > > Reinout, do you have some buildout.cfg scripts you could add in > Distribute, that can be built > quickly and illustrates this problem ? Well, including zc.buildout as a test dependency of distribute is a bit big, I guess. The thing I'd try: add a test that does the thing zc.buildout does (str(exception)) and demonstrate that that's the right answer. Add a comment on why this needs testing. The thing that gave me an error (cmmi through staticlxml) is exactly one of those buildout tasks that takes 15 minutes to compile... ;-) Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets" From ziade.tarek at gmail.com Sat Dec 12 18:34:56 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 18:34:56 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> <4B23BFF9.90705@cryptelium.net> <94bdd2610912120826s6a016977re3fc583424c30ff1@mail.gmail.com> Message-ID: <94bdd2610912120934j6832f36fg46a15d76828fec48@mail.gmail.com> On Sat, Dec 12, 2009 at 6:29 PM, ssteinerX at gmail.com wrote: > > On Dec 12, 2009, at 11:26 AM, Tarek Ziad? wrote: > >> There are hundreds of way to break things :) >> >> IOW everytime a change is made, there's a potential risk. Functional >> tests may help > > Tarek, > > ? ? ? ?What are you using for a test suite pre-release now? I run the test command using python then python3 : $ python setup.py test $ python3 setup.py test the latter triggers the 2to3 script and launch the tests in build/ > > ? ? ? ?I started on the multi-package installer we talked about but we never got back together to get it into the release testing. That would be a good timing to continue then ;) Tarek From ssteinerx at gmail.com Sat Dec 12 18:39:10 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sat, 12 Dec 2009 12:39:10 -0500 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <94bdd2610912120934j6832f36fg46a15d76828fec48@mail.gmail.com> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> <4B23BFF9.90705@cryptelium.net> <94bdd2610912120826s6a016977re3fc583424c30ff1@mail.gmail.com> <94bdd2610912120934j6832f36fg46a15d76828fec48@mail.gmail.com> Message-ID: On Dec 12, 2009, at 12:34 PM, Tarek Ziad? wrote: > On Sat, Dec 12, 2009 at 6:29 PM, ssteinerX at gmail.com > wrote: >> >> On Dec 12, 2009, at 11:26 AM, Tarek Ziad? wrote: >> >>> There are hundreds of way to break things :) >>> >>> IOW everytime a change is made, there's a potential risk. Functional >>> tests may help >> >> Tarek, >> >> What are you using for a test suite pre-release now? > > I run the test command using python then python3 : > > $ python setup.py test > $ python3 setup.py test > > the latter triggers the 2to3 script and launch the tests in build/ > >> >> I started on the multi-package installer we talked about but we never got back together to get it into the release testing. > > > That would be a good timing to continue then ;) I'm thinking you're right... Do we have a buildout that fails to put into the loop? S From reinout at vanrees.org Sat Dec 12 18:35:43 2009 From: reinout at vanrees.org (Reinout van Rees) Date: Sat, 12 Dec 2009 18:35:43 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> Message-ID: On 12/12/09 5:16 PM, Reinout van Rees wrote: > And I'm happy to see that zc.buildout 1.4.3 indeed allows a distribute > upgrade just fine without infinite recursion :-) With the new distribute and buildout release, I've wrote a blog post (http://tinyurl.com/ya5ru3m) to make sure people have something to google when they run into the recursion problem. Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets" From ziade.tarek at gmail.com Sat Dec 12 18:40:49 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 18:40:49 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> Message-ID: <94bdd2610912120940k20d42619w910483c222551632@mail.gmail.com> On Sat, Dec 12, 2009 at 6:33 PM, Reinout van Rees wrote: > On 12/12/09 3:25 PM, Tarek Ziad? wrote: >>>> >>>> AttributeError: 'str' object has no attribute 'project_name' >> >> Ok this is fixed now. 0.6.10 released >> >> I think we should work on adding a functional test in Distribute, that >> runs some zc.buildout scripts >> to make sure this doesn't happen again, >> >> Reinout, do you have some buildout.cfg scripts you could add in >> Distribute, that can be built >> quickly and illustrates this problem ? > > Well, including zc.buildout as a test dependency of distribute is a bit big, > I guess. I wasn't thinking about a regular unit test or functional test, but more like a "smoke" test like what I am doing in distutils. That would be a shell sequence in a script that build a buildout and checks everything went well. > > The thing I'd try: add a test that does the thing zc.buildout does > (str(exception)) and demonstrate that that's the right answer. Add a comment > on why this needs testing. Sure, for that particular case we can add this test, but what I really want is a script that uses Distribute in various execution environments because distribute is just a layer and we can't find such bugs before they happen through other softwares like zc.buildout. > > The thing that gave me an error (cmmi through staticlxml) is exactly one of > those buildout tasks that takes 15 minutes to compile... ;-) Yes, that is why a buildbot can help. Of course we can't test everything, but if we can prevent regressions in zc.buildout, that could be useful. Cheers From ziade.tarek at gmail.com Sat Dec 12 18:42:25 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 18:42:25 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> Message-ID: <94bdd2610912120942s75a6362qec8d7a55dece38a9@mail.gmail.com> On Sat, Dec 12, 2009 at 6:35 PM, Reinout van Rees wrote: > On 12/12/09 5:16 PM, Reinout van Rees wrote: >> >> And I'm happy to see that zc.buildout 1.4.3 indeed allows a distribute >> upgrade just fine without infinite recursion :-) > > With the new distribute and buildout release, I've wrote a blog post > (http://tinyurl.com/ya5ru3m) to make sure people have something to google > when they run into the recursion problem. Thanks for that. From ziade.tarek at gmail.com Sat Dec 12 18:44:12 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 18:44:12 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> <4B23BFF9.90705@cryptelium.net> <94bdd2610912120826s6a016977re3fc583424c30ff1@mail.gmail.com> <94bdd2610912120934j6832f36fg46a15d76828fec48@mail.gmail.com> Message-ID: <94bdd2610912120944m41f3be1asc833ac9fa98fca29@mail.gmail.com> On Sat, Dec 12, 2009 at 6:39 PM, ssteinerX at gmail.com wrote: > > On Dec 12, 2009, at 12:34 PM, Tarek Ziad? wrote: > >> On Sat, Dec 12, 2009 at 6:29 PM, ssteinerX at gmail.com >> wrote: >>> >>> On Dec 12, 2009, at 11:26 AM, Tarek Ziad? wrote: >>> >>>> There are hundreds of way to break things :) >>>> >>>> IOW everytime a change is made, there's a potential risk. Functional >>>> tests may help >>> >>> Tarek, >>> >>> ? ? ? ?What are you using for a test suite pre-release now? >> >> I run the test command using python then python3 : >> >> $ python setup.py test >> $ python3 setup.py test >> >> the latter triggers the 2to3 script and launch the tests in build/ >> >>> >>> ? ? ? ?I started on the multi-package installer we talked about but we never got back together to get it into the release testing. >> >> >> That would be a good timing to continue then ;) > > I'm thinking you're right... > > Do we have a buildout that fails to put into the loop? any buildout that contains a mrdeveloper egg should trigger that particular problem. For regression testing I think using the plone 4 development .cfg could be a good idea because its quite big and complex From ssteinerx at gmail.com Sat Dec 12 18:52:41 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sat, 12 Dec 2009 12:52:41 -0500 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <94bdd2610912120934j6832f36fg46a15d76828fec48@mail.gmail.com> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> <4B23BFF9.90705@cryptelium.net> <94bdd2610912120826s6a016977re3fc583424c30ff1@mail.gmail.com> <94bdd2610912120934j6832f36fg46a15d76828fec48@mail.gmail.com> Message-ID: <73B8EC96-7C5A-4416-8096-A348C7F73CF8@gmail.com> On Dec 12, 2009, at 12:34 PM, Tarek Ziad? wrote: > That would be a good timing to continue then ;) The work I did is in the distutils-buildbot branch "smoke_with_mirrors". Basically, what I did was set up a configuration file that contained the packages we were going to install, what versions they are to be installed on. What still needs to be sone is to iterate through that configuration, doing the actual install/run test suite. Could you take a quick look and see if you agree with the approach I took? Thanks, S From ziade.tarek at gmail.com Sat Dec 12 19:27:03 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 12 Dec 2009 19:27:03 +0100 Subject: [Distutils] Problem with buildout and zc.recipe.cmmi In-Reply-To: <73B8EC96-7C5A-4416-8096-A348C7F73CF8@gmail.com> References: <4B22F984.50107@pacbell.net> <94bdd2610912120606o5a3b0343p9c4cd89019888d05@mail.gmail.com> <94bdd2610912120625h513a70d2v50d9365eabf9fd99@mail.gmail.com> <4B23BFF9.90705@cryptelium.net> <94bdd2610912120826s6a016977re3fc583424c30ff1@mail.gmail.com> <94bdd2610912120934j6832f36fg46a15d76828fec48@mail.gmail.com> <73B8EC96-7C5A-4416-8096-A348C7F73CF8@gmail.com> Message-ID: <94bdd2610912121027g2b95adaaqe614c1d4b8c53f69@mail.gmail.com> On Sat, Dec 12, 2009 at 6:52 PM, ssteinerX at gmail.com wrote: > > On Dec 12, 2009, at 12:34 PM, Tarek Ziad? wrote: > >> That would be a good timing to continue then ;) > > The work I did is in the distutils-buildbot branch "smoke_with_mirrors". > > Basically, what I did was set up a configuration file that contained the packages we were going to install, what versions they are to be installed on. > > What still needs to be sone is to iterate through that configuration, doing the actual install/run test suite. > > Could you take a quick look and see if you agree with the approach I took? > It looks good to me, it makes it simpler to configure > Thanks, > > S > > -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From LuHe at gmx.at Sun Dec 13 15:08:36 2009 From: LuHe at gmx.at (Lukas Hetzenecker) Date: Sun, 13 Dec 2009 15:08:36 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder Message-ID: <200912131508.36740.LuHe@gmx.at> Hello, i have the following folder structure: INSTALL Changelog LICENCE main.py module.py modile.py Is it possible to create a setup.py script which copies INSTALL, Changelog and LICENCE also in the installation folder (by default /usr/lib/python2.6/site- packages/application)? I have this script: from distutils.core import setup setup(name='application', version='1.0', packages=['src', 'src.module1', 'src.module2'], package_data={'' : ['Changelog', 'HACKING', 'INSTALL', 'LICENSE', 'LICENSE.icons-oxygen', 'README.icons-oxygen', 'TODO']}, scripts=['application-wrapper'] ) I also tried it so: from distutils.core import setup setup(name='application', version='1.0', packages=['application', 'application.module1', 'application.module2'], package_dir={'application': 'src/'}, package_data={'application' : ['Changelog', 'HACKING', 'INSTALL', 'LICENSE', 'LICENSE.icons-oxygen', 'README.icons-oxygen', 'TODO']}, scripts=['application-wrapper'] ) But this did also not work. I think another way would be to use data_files instead of package_data - but this copys the files to /usr/bin by default, and I don't know the actual application directory - or would this be possible? (If you suggest to move the content of the src/ directory to the root folder, or to copy the INSTALL and LICENCE files to the src folder, this wouldn't work. The src folder is actually not named src/ but pc/ and there is also a folder for the part of the mobile phone - and I want the LICENSE and INSTALL file in the root directory to not confuse the users, but I only want to install the content of the pc/ (or as in this simple example, the src/ folder) on the computer.) Thanks for any help, Lukas From ziade.tarek at gmail.com Sun Dec 13 15:27:29 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 13 Dec 2009 15:27:29 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <200912131508.36740.LuHe@gmx.at> References: <200912131508.36740.LuHe@gmx.at> Message-ID: <94bdd2610912130627l210e4811y56000fc126f895a@mail.gmail.com> On Sun, Dec 13, 2009 at 3:08 PM, Lukas Hetzenecker wrote: > Hello, > > i have the following folder structure: Hi Lukas, your project layout is a bit exotic but looks fine for what you want to do. You just need to add an extra file called MANIFEST.in alongside setup.py, with this content: recursive-include application *.py include application/INSTALL include application/Changelog include application/LICENCE This is a template file, that'll make sure package_data files are included in the archive then installed. Notice that this has changed in version 2.7: All the files that match package_data will be added to the MANIFEST file if no template is provided. IOW this MANIFEST.in won't be necessary anymore in your case in the future see http://docs.python.org/dev/distutils/sourcedist.html#manifest Tarek From LuHe at gmx.at Sun Dec 13 19:42:10 2009 From: LuHe at gmx.at (Lukas Hetzenecker) Date: Sun, 13 Dec 2009 19:42:10 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <94bdd2610912130627l210e4811y56000fc126f895a@mail.gmail.com> References: <200912131508.36740.LuHe@gmx.at> <94bdd2610912130627l210e4811y56000fc126f895a@mail.gmail.com> Message-ID: <200912131942.10826.LuHe@gmx.at> Hello, thanks for your reply. I added these files to MANIFEST.in but they are only in the tarball (generated using python setup.py sdist), and not in the installation directory. I attached the tree of my source (source_tree), my setup.py script, my MANIFEST.in and the tree of the installation directory. Thanks, Lukas Am Sonntag 13 Dezember 2009 15:27:29 schrieben Sie: > On Sun, Dec 13, 2009 at 3:08 PM, Lukas Hetzenecker wrote: > > Hello, > > > > i have the following folder structure: > > Hi Lukas, your project layout is a bit exotic but looks fine for what > you want to do. > > You just need to add an extra file called MANIFEST.in alongside > setup.py, with this content: > > recursive-include application *.py > include application/INSTALL > include application/Changelog > include application/LICENCE > > This is a template file, that'll make sure package_data files are > included in the archive then installed. > > Notice that this has changed in version 2.7: All the files that match > package_data will be added to the MANIFEST file if no template is > provided. > > IOW this MANIFEST.in won't be necessary anymore in your case in the future > > see http://docs.python.org/dev/distutils/sourcedist.html#manifest > > Tarek > -------------- next part -------------- root at laptop /usr/lib/python2.6/site-packages # tree --dirsfirst series60_remote |grep -v ".svn" |grep -v ".png" |grep -v ".pyc" |grep -v ".ui" |grep -v ".svg" > directory_tree series60_remote ??? devices ??? ??? __init__.py ??? ??? null.py ??? ??? series60.py ??? ??? status_numbers.py ??? lib ??? ??? classes.py ??? ??? database.py ??? ??? dbusobject.py ??? ??? delegate_movie.py ??? ??? device.py ??? ??? favorites.py ??? ??? helper.py ??? ??? import_messages.py ??? ??? __init__.py ??? ??? log.py ??? ??? obex_completer.py ??? ??? obex_handler.py ??? ??? obex_iconprovider.py ??? ??? obex_items.py ??? ??? obex_model.py ??? ??? obex_scheduler.py ??? ??? obex_wrapper.py ??? ??? settings.py ??? ??? __init__.py ??? ??? resource_rc.py ??? widget ??? ??? ContactCanvas.py ??? ??? Delegate.py ??? ??? ImageButton.py ??? ??? __init__.py ??? ??? MessageTextEdit.py ??? ??? ObexView.py ??? ??? ObexWidget.py ??? ??? PixmapRegionSelectorWidget.py ??? ??? Popup.py ??? ??? SearchLineEdit.py ??? ??? StatisticCanvas.py ??? window ??? ??? about.py ??? ??? chat.py ??? ??? contacts_edit.py ??? ??? favorites.py ??? ??? history.py ??? ??? import_messages.py ??? ??? __init__.py ??? ??? log.py ??? ??? mainwindow.py ??? ??? message_queue.py ??? ??? pixmap_region_selector_dialog.py ??? ??? popups.py ??? ??? settings.py ??? ??? statistics.py ??? ??? systemtrayicon.py ??? ??? wizard.py ??? __init__.py ??? mkpyqt.py ??? series60_remote.py 5 directories, 158 files -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py Type: text/x-python Size: 538 bytes Desc: not available URL: -------------- next part -------------- lukas at laptop /home/lukas/Projekte/series60-remote/trunk (r390) # tree --dirsfirst |grep -v ".svn" |grep -v ".png" |grep -v ".pyc" |grep -v ".ui" |grep -v ".sv > directory_tree . ??? lib ??? mobile ??? ??? create_package ??? ??? __init__.py ??? ??? mobile.py ??? ??? PythonForS60_1_4_5_3rdEd.sis ??? ??? series60-remote.sis ??? pc ??? ??? devices ??? ??? ??? __init__.py ??? ??? ??? null.py ??? ??? ??? series60.py ??? ??? ??? status_numbers.py ??? ??? lang ??? ??? ??? app_de.qm ??? ??? ??? app_de.ts ??? ??? ??? qt_de.qm ??? ??? lib ??? ??? ??? classes.py ??? ??? ??? database.py ??? ??? ??? dbusobject.py ??? ??? ??? delegate_movie.py ??? ??? ??? device.py ??? ??? ??? favorites.py ??? ??? ??? helper.py ??? ??? ??? import_messages.py ??? ??? ??? __init__.py ??? ??? ??? log.py ??? ??? ??? obex_completer.py ??? ??? ??? obex_handler.py ??? ??? ??? obex_iconprovider.py ??? ??? ??? obex_items.py ??? ??? ??? obex_model.py ??? ??? ??? obex_scheduler.py ??? ??? ??? obex_wrapper.py ??? ??? ??? settings.py ??? ??? pics ??? ??? ??? hicolor ??? ??? ??? ??? scalable ??? ??? ??? other ??? ??? ??? oxygen ??? ??? ??? ??? scalable ??? ??? ??? ??? ??? actions ??? ??? ??? ??? ??? apps ??? ??? ??? ??? ??? devices ??? ??? ??? ??? ??? mimetypes ??? ??? ??? ??? ??? places ??? ??? ??? ??? ??? status ??? ??? ??? ??? LICENSE ??? ??? ??? ??? README ??? ??? ??? oxygen-icon-list ??? ??? ??? phone.ico ??? ??? ??? update-oxygen.py ??? ??? ??? src ??? ??? ??? ??? resource.qrc ??? ??? ??? __init__.py ??? ??? ??? resource_rc.py ??? ??? widget ??? ??? ??? ContactCanvas.py ??? ??? ??? Delegate.py ??? ??? ??? ImageButton.py ??? ??? ??? __init__.py ??? ??? ??? MessageTextEdit.py ??? ??? ??? ObexView.py ??? ??? ??? ObexWidget.py ??? ??? ??? PixmapRegionSelectorWidget.py ??? ??? ??? Popup.py ??? ??? ??? SearchLineEdit.py ??? ??? ??? StatisticCanvas.py ??? ??? window ??? ??? ??? about.py ??? ??? ??? chat.py ??? ??? ??? contacts_edit.py ??? ??? ??? favorites.py ??? ??? ??? history.py ??? ??? ??? import_messages.py ??? ??? ??? __init__.py ??? ??? ??? log.py ??? ??? ??? mainwindow.py ??? ??? ??? message_queue.py ??? ??? ??? pixmap_region_selector_dialog.py ??? ??? ??? popups.py ??? ??? ??? settings.py ??? ??? ??? statistics.py ??? ??? ??? systemtrayicon.py ??? ??? ??? wizard.py ??? ??? generate-pro.sh ??? ??? __init__.py ??? ??? mkpyqt.py ??? ??? send_to_mobile ??? ??? series60-remote.coverage ??? ??? series60-remote.pro ??? ??? series60_remote.py ??? series60_remote.egg-info ??? ??? dependency_links.txt ??? ??? PKG-INFO ??? ??? SOURCES.txt ??? ??? top_level.txt ??? Changelog ??? directory_tree ??? HACKING ??? __init__.py ??? INSTALL ??? LICENSE ??? LICENSE.icons-oxygen ??? MANIFEST.in ??? README.icons-oxygen ??? series60-remote ??? series60-remote.e4p ??? setup.py ??? TODO 23 directories, 328 files -------------- next part -------------- include INSTALL include Changelog include LICENSE include LICENSE.icons-oxygen include README.icons-oxygen include TODO include HACKING From cournape at gmail.com Mon Dec 14 05:40:34 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 14 Dec 2009 10:10:34 +0530 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <200912131942.10826.LuHe@gmx.at> References: <200912131508.36740.LuHe@gmx.at> <94bdd2610912130627l210e4811y56000fc126f895a@mail.gmail.com> <200912131942.10826.LuHe@gmx.at> Message-ID: <5b8d13220912132040l4830cc6fna00aed440ba3d8b0@mail.gmail.com> On Mon, Dec 14, 2009 at 12:12 AM, Lukas Hetzenecker wrote: > Hello, > > thanks for your reply. > I added these files to MANIFEST.in but they are only in the tarball (generated > using python setup.py sdist), and not in the installation directory. Yes, MANIFEST.in does not handle installed files, and is only concerned with sdist (the MANIFEST.in handling is done in the sdist command). There is no easy solution to your problem AFAIK. Since your data files are outside a package, you have to use data_files argument, but data_files arguments only install relatively to root or the install prefix. You need to write your own command for this, at least with straight distutils. The easier solution is to put INSTALL and co into a package, and use package_data: from distutils.core import setup if __name__ == '__main__': dist = setup( name='example', packages=['foo'], package_data={'foo': ['INSTALL']}, ) In this case, INSTALL will be installed wherever the package foo is installed, but this requires INSTALL to be in the foo package. David From LuHe at gmx.at Mon Dec 14 12:12:51 2009 From: LuHe at gmx.at (Lukas Hetzenecker) Date: Mon, 14 Dec 2009 12:12:51 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <5b8d13220912132040l4830cc6fna00aed440ba3d8b0@mail.gmail.com> References: <200912131508.36740.LuHe@gmx.at> <200912131942.10826.LuHe@gmx.at> <5b8d13220912132040l4830cc6fna00aed440ba3d8b0@mail.gmail.com> Message-ID: <200912141212.52046.LuHe@gmx.at> Hello, i just found this solution and is seems to work for my application: import os from distutils.core import setup # Install data files to package directory from distutils.command.install import INSTALL_SCHEMES for scheme in INSTALL_SCHEMES.values(): scheme['data'] = scheme['purelib'] setup(name='series60-remote', version='1.0', packages=['series60_remote', 'series60_remote.devices', 'series60_remote.lib', 'series60_remote.ui', 'series60_remote.window', 'series60_remote.widget'], package_dir={'series60_remote': 'pc'}, data_files=[('series60_remote', ['Changelog', 'HACKING', 'INSTALL', 'LICENSE', 'LICENSE.icons-oxygen', 'README.icons-oxygen', 'TODO']), scripts=['series60-remote'], ) Am Montag 14 Dezember 2009 05:40:34 schrieb David Cournapeau: > On Mon, Dec 14, 2009 at 12:12 AM, Lukas Hetzenecker wrote: > > Hello, > > > > thanks for your reply. > > I added these files to MANIFEST.in but they are only in the tarball > > (generated using python setup.py sdist), and not in the installation > > directory. > > Yes, MANIFEST.in does not handle installed files, and is only > concerned with sdist (the MANIFEST.in handling is done in the sdist > command). > > There is no easy solution to your problem AFAIK. Since your data files > are outside a package, you have to use data_files argument, but > data_files arguments only install relatively to root or the install > prefix. > > You need to write your own command for this, at least with straight > distutils. The easier solution is to put INSTALL and co into a > package, and use package_data: > > from distutils.core import setup > > if __name__ == '__main__': > dist = setup( > name='example', > packages=['foo'], > package_data={'foo': ['INSTALL']}, > ) > > In this case, INSTALL will be installed wherever the package foo is > installed, but this requires INSTALL to be in the foo package. > > David > From ziade.tarek at gmail.com Mon Dec 14 12:19:36 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 14 Dec 2009 12:19:36 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <5b8d13220912132040l4830cc6fna00aed440ba3d8b0@mail.gmail.com> References: <200912131508.36740.LuHe@gmx.at> <94bdd2610912130627l210e4811y56000fc126f895a@mail.gmail.com> <200912131942.10826.LuHe@gmx.at> <5b8d13220912132040l4830cc6fna00aed440ba3d8b0@mail.gmail.com> Message-ID: <94bdd2610912140319l5864985bwe740f90bb1d8293b@mail.gmail.com> 2009/12/14 David Cournapeau : > On Mon, Dec 14, 2009 at 12:12 AM, Lukas Hetzenecker wrote: >> Hello, >> >> thanks for your reply. >> I added these files to MANIFEST.in but they are only in the tarball (generated >> using python setup.py sdist), and not in the installation directory. > > Yes, MANIFEST.in does not handle installed files, and is only > concerned with sdist (the MANIFEST.in handling is done in the sdist > command). Sorry I've misread Lukas setup.py, I thought "application" was a package. From ziade.tarek at gmail.com Mon Dec 14 12:30:20 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 14 Dec 2009 12:30:20 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <200912141212.52046.LuHe@gmx.at> References: <200912131508.36740.LuHe@gmx.at> <200912131942.10826.LuHe@gmx.at> <5b8d13220912132040l4830cc6fna00aed440ba3d8b0@mail.gmail.com> <200912141212.52046.LuHe@gmx.at> Message-ID: <94bdd2610912140330n3904a8e6q78fc519dcbe2160d@mail.gmail.com> 2009/12/14 Lukas Hetzenecker : > Hello, > > i just found this solution and is seems to work for my application: > > import os > from distutils.core import setup > > # Install data files to package directory > from distutils.command.install import INSTALL_SCHEMES > for scheme in INSTALL_SCHEMES.values(): > ? ?scheme['data'] = scheme['purelib'] Hacking the schemes that way can get you in trouble if the end user system runs a command that work with the schemes. Another clean way to achieve this is to copy those files temporarely in the pc directory in your release process, so your files will be in a *package* declared in packages and then you can use package_data together with MANIFEST.in: http://docs.python.org/distutils/setupscript.html#installing-package-data Tarek From LuHe at gmx.at Mon Dec 14 12:56:41 2009 From: LuHe at gmx.at (Lukas Hetzenecker) Date: Mon, 14 Dec 2009 12:56:41 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <94bdd2610912140330n3904a8e6q78fc519dcbe2160d@mail.gmail.com> References: <200912131508.36740.LuHe@gmx.at> <200912141212.52046.LuHe@gmx.at> <94bdd2610912140330n3904a8e6q78fc519dcbe2160d@mail.gmail.com> Message-ID: <200912141256.41453.LuHe@gmx.at> Hello, so you think that this is a better way: import os from distutils.core import setup from distutils.file_util import copy_file files = ['Changelog', 'HACKING', 'INSTALL', 'LICENSE', 'LICENSE.icons-oxygen', 'README.icons-oxygen', 'TODO'] # Copy files temporarely to pc directory for file in files: copy_file(file, "pc/") pys60 = 'PythonForS60_1_4_5_3rdEd.sis' if os.name == 'posix': sisfolder = "/usr/share/series60-remote/" else: sisfolder = "mobile/" setup(name='series60-remote', version='1.0', packages=['series60_remote', 'series60_remote.devices', 'series60_remote.lib', 'series60_remote.ui', 'series60_remote.window', 'series60_remote.widget'], package_dir={'series60_remote': 'pc'}, package_data={'series60_remote' : files}, data_files=[(sisfolder, ['mobile/series60-remote.sis', 'mobile/' + pys60])], scripts=['series60-remote'], ) # Remove temp files for file in files: os.remove("pc/" + file) Thank you for all your comments, Lukas Am Montag 14 Dezember 2009 12:30:20 schrieben Sie: > 2009/12/14 Lukas Hetzenecker : > > Hello, > > > > i just found this solution and is seems to work for my application: > > > > import os > > from distutils.core import setup > > > > # Install data files to package directory > > from distutils.command.install import INSTALL_SCHEMES > > for scheme in INSTALL_SCHEMES.values(): > > scheme['data'] = scheme['purelib'] > > Hacking the schemes that way can get you in trouble if the end user system > runs a command that work with the schemes. > > Another clean way to achieve this is to copy those files temporarely > in the pc directory > in your release process, so your files will be in a *package* declared > in packages and then you can use package_data together with > MANIFEST.in: > > http://docs.python.org/distutils/setupscript.html#installing-package-data > > Tarek > From LuHe at gmx.at Mon Dec 14 13:33:32 2009 From: LuHe at gmx.at (Lukas Hetzenecker) Date: Mon, 14 Dec 2009 13:33:32 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <200912141256.41453.LuHe@gmx.at> References: <200912131508.36740.LuHe@gmx.at> <94bdd2610912140330n3904a8e6q78fc519dcbe2160d@mail.gmail.com> <200912141256.41453.LuHe@gmx.at> Message-ID: <200912141333.32606.LuHe@gmx.at> Sorry, this version didn't run correctly on windows, i attached a new one. It would be great if you could quickly look at it and tell me if this could make any problems. Thanks Lukas Am Montag 14 Dezember 2009 12:56:41 schrieb Lukas Hetzenecker: > Hello, > > so you think that this is a better way: > > import os > from distutils.core import setup > from distutils.file_util import copy_file > > files = ['Changelog', 'HACKING', 'INSTALL', 'LICENSE', > 'LICENSE.icons-oxygen', 'README.icons-oxygen', 'TODO'] > > # Copy files temporarely to pc directory > for file in files: > copy_file(file, "pc/") > > pys60 = 'PythonForS60_1_4_5_3rdEd.sis' > > if os.name == 'posix': > sisfolder = "/usr/share/series60-remote/" > else: > sisfolder = "mobile/" > > setup(name='series60-remote', > version='1.0', > packages=['series60_remote', 'series60_remote.devices', > 'series60_remote.lib', 'series60_remote.ui', > 'series60_remote.window', 'series60_remote.widget'], > package_dir={'series60_remote': 'pc'}, > package_data={'series60_remote' : files}, > data_files=[(sisfolder, ['mobile/series60-remote.sis', 'mobile/' + > pys60])], > scripts=['series60-remote'], > ) > > # Remove temp files > for file in files: > os.remove("pc/" + file) > > > > Thank you for all your comments, > Lukas > > Am Montag 14 Dezember 2009 12:30:20 schrieben Sie: > > 2009/12/14 Lukas Hetzenecker : > > > Hello, > > > > > > i just found this solution and is seems to work for my application: > > > > > > import os > > > from distutils.core import setup > > > > > > # Install data files to package directory > > > from distutils.command.install import INSTALL_SCHEMES > > > for scheme in INSTALL_SCHEMES.values(): > > > scheme['data'] = scheme['purelib'] > > > > Hacking the schemes that way can get you in trouble if the end user > > system runs a command that work with the schemes. > > > > Another clean way to achieve this is to copy those files temporarely > > in the pc directory > > in your release process, so your files will be in a *package* declared > > in packages and then you can use package_data together with > > MANIFEST.in: > > > > http://docs.python.org/distutils/setupscript.html#installing-package-data > > > > Tarek > -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py Type: text/x-python Size: 1274 bytes Desc: not available URL: From chris at simplistix.co.uk Mon Dec 14 15:05:02 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 14 Dec 2009 14:05:02 +0000 Subject: [Distutils] [buildout] "Couldn't find index page for" in tests Message-ID: <4B26460E.6040305@simplistix.co.uk> Hi All, I see a few tests for buildout recipes that use zc.buildout.testing but also use a RENormalizer to hide "Couldn't find index page for" errors.. Why is this error showing up and why is it okay to hide it? not-joining-the-cargo-cult-ly yours, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Mon Dec 14 15:44:54 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 14 Dec 2009 15:44:54 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <200912141333.32606.LuHe@gmx.at> References: <200912131508.36740.LuHe@gmx.at> <94bdd2610912140330n3904a8e6q78fc519dcbe2160d@mail.gmail.com> <200912141256.41453.LuHe@gmx.at> <200912141333.32606.LuHe@gmx.at> Message-ID: <94bdd2610912140644t15b3e1fbg9f2a57d8185afc4b@mail.gmail.com> On Mon, Dec 14, 2009 at 1:33 PM, Lukas Hetzenecker wrote: > Sorry, this version didn't run correctly on windows, i attached a new one. > It would be great if you could quickly look at it and tell me if this could > make any problems. What I meant was to copy your files in the package directory when you create your distribution, *before* "python setup.py sdist" is called, so the classical install mechanisms will work. e.g.= relase.sh: cp file pc/file python setup.py sdist rm pc/file Meaning that in your development layout, they would stay out of "pc", but get in there when you create the archive. What you have done will not work as-is because setup.py can be called for doing something else than 'sdist', so you need to run the code only if 'sdist' is called. But fixing this can make your setup.py code more fragile, thats why I was thinking of a higher level script. Last, if you really dislike the idea of seeing these file shipped in the distribution archive within a package, (pc), David's proposal is the less hackish: Create your own distutils command with your custom behavior for copying files in the target system at install time. overriding install_data.run would be the simplest way in that case. see: http://docs.python.org/distutils/extending.html#integrating-new-commands Tarek From ssteinerx at gmail.com Mon Dec 14 15:47:25 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Mon, 14 Dec 2009 09:47:25 -0500 Subject: [Distutils] [buildout] "Couldn't find index page for" in tests In-Reply-To: <4B26460E.6040305@simplistix.co.uk> References: <4B26460E.6040305@simplistix.co.uk> Message-ID: <2B29A4BD-5600-498B-8084-4FE8DC847332@gmail.com> On Dec 14, 2009, at 9:05 AM, Chris Withers wrote: > Hi All, > > I see a few tests for buildout recipes that use zc.buildout.testing but also use a RENormalizer to hide "Couldn't find index page for" errors.. > > Why is this error showing up and why is it okay to hide it? > > not-joining-the-cargo-cult-ly yours, Which cargo cult are you not joining? Perhaps we could start a news/support group. S From LuHe at gmx.at Mon Dec 14 16:49:42 2009 From: LuHe at gmx.at (Lukas Hetzenecker) Date: Mon, 14 Dec 2009 16:49:42 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <94bdd2610912140644t15b3e1fbg9f2a57d8185afc4b@mail.gmail.com> References: <200912131508.36740.LuHe@gmx.at> <200912141333.32606.LuHe@gmx.at> <94bdd2610912140644t15b3e1fbg9f2a57d8185afc4b@mail.gmail.com> Message-ID: <200912141649.42768.LuHe@gmx.at> Am Montag 14 Dezember 2009 15:44:54 schrieben Sie: > What I meant was to copy your files in the package directory when you > create your distribution, > *before* "python setup.py sdist" is called, so the classical install > mechanisms will work. > e.g.= > > relase.sh: > cp file pc/file > python setup.py sdist > rm pc/file > > Meaning that in your development layout, they would stay out of "pc", > but get in there when you create the archive. The layout for the release tarball should be like my development layout, only the installation layout should be different. This means: The user downloads a tarball which has the package for his pc (in the pc/ directory), a package for his mobile phone (in the mobile/ directory) and the LICENCE, README files (in the root directory). Now he wants to install the pc package. Only now the modules of the pc/ directory are mixed with the README files and copied to the site-packages directory. So he has the README files in the root directory of his tarball _and_ in the package (site-packages/series60_remote) directory when he installed the script. > What you have done will not work as-is because setup.py can be called > for doing something else than 'sdist', so you need to run the code > only if 'sdist' is called. Well, this "mixing" should _not_ happen when sdist is called. > But fixing this can make your setup.py code more fragile, thats why I > was thinking of a higher level > script. > > Last, if you really dislike the idea of seeing these file shipped in > the distribution archive within a package, (pc), David's proposal is > the less hackish: > > Create your own distutils command with your custom behavior for > copying files in the target system > at install time. overriding install_data.run would be the simplest way > in that case. > > see: > http://docs.python.org/distutils/extending.html#integrating-new-commands > > > Tarek > From reinout at vanrees.org Mon Dec 14 17:59:16 2009 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 14 Dec 2009 17:59:16 +0100 Subject: [Distutils] [buildout] "Couldn't find index page for" in tests In-Reply-To: <4B26460E.6040305@simplistix.co.uk> References: <4B26460E.6040305@simplistix.co.uk> Message-ID: On 12/14/09 3:05 PM, Chris Withers wrote: > Hi All, > > I see a few tests for buildout recipes that use zc.buildout.testing but > also use a RENormalizer to hide "Couldn't find index page for" errors.. > > Why is this error showing up and why is it okay to hide it? I saw it too lately when working on z3c.recipe.compattest. I thought the "index page not found" meant that you did not add those packages to your test buildout environment with either zc.buildout.testing.install() or zc.buildout.testing.install_develop(), see for instance http://svn.plone.org/svn/collective/collective.buildbot/trunk/collective/buildbot/tests/test_docs.py Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets" From jorge.vargas at gmail.com Tue Dec 15 00:26:02 2009 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Mon, 14 Dec 2009 17:26:02 -0600 Subject: [Distutils] Should pip support extras? Was: [venv] question about future pip capabilities Message-ID: <32822fe60912141526n481b421cndee01dacb5ac139b@mail.gmail.com> On Thu, Dec 10, 2009 at 3:58 PM, Darren Dale wrote: > On Thu, Dec 10, 2009 at 2:30 PM, Ian Bicking wrote: >> On Thu, Dec 10, 2009 at 11:18 AM, Darren Dale wrote: >> There isn't really; extras would be nice, but there hasn't been much >> demand for them. ?pip requirement files serve a similar purpose though >> in a somewhat different way. > > I brought this up on the virtualenv mailing list once before. > Requirements files serve a very different use-case. If I want to > define a project extra like foo[traits_qt4] that depends on a project > with an extra like traits[qt4] > 3.2.0, I can't just list traits[qt4] >> 3.2.0 in my requirements file, because pip is not designed to > recursively check the (arbitrarily-named) requirements files of the > dependencies I declare in my own requirements file. I have to keep > track of not only my projects dependencies, but my dependencies > dependencies, and on and on. > I think the problem with extras is that the concept itself is flawed. python lacks this optional dependecies paradigm and we end up with a bunch of hacks to get the optional dependencies to work. And in the end you get to maintain code that checks at runtime which "option" to choose making the extras stuff redundant at best. I can't really think of a case where pip install lib backend1 pip install lib backend2 can't replace pip install lib[backend1] pip install lib[backend2] as for requirements being a replacement what's wrong with having several files? sure it's a little duplication (and it may get exponential if you depend on a package that depends on other optional dependencies) but seriously which % of package suffer from that? IMO extras was a misfeature that should be slowly deprecated. > Regards, > Darren > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From dsdale24 at gmail.com Tue Dec 15 01:30:05 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 14 Dec 2009 19:30:05 -0500 Subject: [Distutils] Should pip support extras? Was: [venv] question about future pip capabilities In-Reply-To: <32822fe60912141526n481b421cndee01dacb5ac139b@mail.gmail.com> References: <32822fe60912141526n481b421cndee01dacb5ac139b@mail.gmail.com> Message-ID: On Mon, Dec 14, 2009 at 6:26 PM, Jorge Vargas wrote: > On Thu, Dec 10, 2009 at 3:58 PM, Darren Dale wrote: >> On Thu, Dec 10, 2009 at 2:30 PM, Ian Bicking wrote: >>> On Thu, Dec 10, 2009 at 11:18 AM, Darren Dale wrote: > >>> There isn't really; extras would be nice, but there hasn't been much >>> demand for them. ?pip requirement files serve a similar purpose though >>> in a somewhat different way. >> >> I brought this up on the virtualenv mailing list once before. >> Requirements files serve a very different use-case. If I want to >> define a project extra like foo[traits_qt4] that depends on a project >> with an extra like traits[qt4] > 3.2.0, I can't just list traits[qt4] >>> 3.2.0 in my requirements file, because pip is not designed to >> recursively check the (arbitrarily-named) requirements files of the >> dependencies I declare in my own requirements file. I have to keep >> track of not only my projects dependencies, but my dependencies >> dependencies, and on and on. >> > I think the problem with extras is that the concept itself is flawed. > python lacks this optional dependecies paradigm and we end up with a > bunch of hacks to get the optional dependencies to work. And in the > end you get to maintain code that checks at runtime which "option" to > choose making the extras stuff redundant at best. I can't really think > of a case where > pip install lib backend1 > pip install lib backend2 > > can't replace > > pip install lib[backend1] > pip install lib[backend2] Maybe. I like what Enthought has done with extras. > as for requirements being a replacement what's wrong with having > several files? Lack of recursive dependency checking. What if a new version of my dependency is released, the old one is no longer available, and the new version has a new dependency? My requirements file won't install that dependency. > sure it's a little duplication (and it may get > exponential if you depend on a package that depends on other optional > dependencies) but seriously which % of package suffer from that? > > IMO extras was a misfeature that should be slowly deprecated. This is why I think talk of deprecating easy_install is premature. I value project extras, I know of other projects that value them as well. I value the ability to declare my projects' immediate dependencies and have those dependencies manage their own dependencies as well. Maybe these details should be worked out, and pip can demonstrate that it meets the needs of the community in practice before planning to deprecate easy_install. Darren From cournape at gmail.com Tue Dec 15 08:28:16 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 15 Dec 2009 12:58:16 +0530 Subject: [Distutils] install_requires vs requires Message-ID: <5b8d13220912142328ydb6823fg3c7bd26b2f005cc3@mail.gmail.com> Hi, I am working on a new packaging solution, and I want to convert existing setup.py. I noticed that the install_requires as used by setuptools is not put into PKG-INFO as requires. Is this expected ? What is the difference between requires and require_install ? What happens when both have different values ? cheers, David From ziade.tarek at gmail.com Tue Dec 15 10:10:46 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 15 Dec 2009 10:10:46 +0100 Subject: [Distutils] install_requires vs requires In-Reply-To: <5b8d13220912142328ydb6823fg3c7bd26b2f005cc3@mail.gmail.com> References: <5b8d13220912142328ydb6823fg3c7bd26b2f005cc3@mail.gmail.com> Message-ID: <94bdd2610912150110q64dbf5duc307e9a4e0d4139f@mail.gmail.com> On Tue, Dec 15, 2009 at 8:28 AM, David Cournapeau wrote: > Hi, > > I am working on a new packaging solution, and I want to convert > existing setup.py. I noticed that the install_requires as used by > setuptools is not put into PKG-INFO as requires. Is this expected ? Yes because it is not part of the standard, so setuptools is putting this in "requires.txt" which could be considered as an extension to the metadata. PEP 345 will include a new field that can replace requires.txt. > What is the difference between requires and require_install ? What > happens when both have different values ? What are those names ? See http://packages.python.org/distribute/setuptools.html#new-and-changed-setup-keywords for the list of new keywords introduced by Setuptools (+ a few more by Distribute) > cheers, > > David > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From andreev at iki.fi Tue Dec 15 17:17:26 2009 From: andreev at iki.fi (Andrey Andreev) Date: Tue, 15 Dec 2009 17:17:26 +0100 Subject: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe Message-ID: <4B27B696.2050904@iki.fi> Hi Rune, I had a similar issue on Windows Server 2008 R2. You are probably running Win 7 (64bit). I ran procmon, and found out that the installer is looking for the keys under: HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore HKEY_CURRENT_USER\Software\Python\PythonCore so I modified the register.py discussed above to write to HKEY_CURRENT_USER instead of HKEY_LOCAL_MACHINE, which solved the issue. Cheers, Andrey From chris at simplistix.co.uk Tue Dec 15 19:30:18 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 15 Dec 2009 18:30:18 +0000 Subject: [Distutils] [buildout] "Couldn't find index page for" in tests In-Reply-To: References: <4B26460E.6040305@simplistix.co.uk> Message-ID: <4B27D5BA.8020102@simplistix.co.uk> Reinout van Rees wrote: > I thought the "index page not found" meant that you did not add those > packages to your test buildout environment with either > zc.buildout.testing.install() or zc.buildout.testing.install_develop(), Nope, if you don't do these, you get errors saying the package couldn't be installed. So, still looking for the answer... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From pje at telecommunity.com Tue Dec 15 22:13:21 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 15 Dec 2009 16:13:21 -0500 Subject: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe In-Reply-To: <4B27B696.2050904@iki.fi> References: <4B27B696.2050904@iki.fi> Message-ID: <20091215211338.2B0A23A408B@sparrow.telecommunity.com> At 05:17 PM 12/15/2009 +0100, Andrey Andreev wrote: >Hi Rune, > >I had a similar issue on Windows Server 2008 R2. You are probably >running Win 7 (64bit). I ran procmon, and found out that the >installer is looking for the keys under: > >HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore >HKEY_CURRENT_USER\Software\Python\PythonCore > >so I modified the register.py discussed above to write to >HKEY_CURRENT_USER instead of HKEY_LOCAL_MACHINE, which solved the issue. Sounds like maybe there's a problem with either the distutils or the Windows Python installer, since either the bdist_wininst isn't checking in the right place, or the Windows Python installer isn't registering in the right place. (i.e., this would be an issue with *any* Python .exe-based installer on these platforms.) From andreev at iki.fi Tue Dec 15 22:39:02 2009 From: andreev at iki.fi (Andrey Andreev) Date: Tue, 15 Dec 2009 22:39:02 +0100 Subject: [Distutils] [Bug] Failed installation. setuptools-0.6c11.win32-py2.6.exe In-Reply-To: <20091215211338.2B0A23A408B@sparrow.telecommunity.com> References: <4B27B696.2050904@iki.fi> <20091215211338.2B0A23A408B@sparrow.telecommunity.com> Message-ID: <4B2801F6.6040903@iki.fi> Hi, P.J. Eby wrote: > At 05:17 PM 12/15/2009 +0100, Andrey Andreev wrote: >> I had a similar issue on Windows Server 2008 R2. You are probably >> running Win 7 (64bit). I ran procmon, and found out that the installer >> is looking for the keys under: >> >> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore >> HKEY_CURRENT_USER\Software\Python\PythonCore >> >> so I modified the register.py discussed above to write to >> HKEY_CURRENT_USER instead of HKEY_LOCAL_MACHINE, which solved the issue. > > Sounds like maybe there's a problem with either the distutils or the > Windows Python installer, since either the bdist_wininst isn't checking > in the right place, or the Windows Python installer isn't registering in > the right place. (i.e., this would be an issue with *any* Python > .exe-based installer on these platforms.) To make it more specific, my configuration is 64bit Python on 64bit OS. Therefore it registers under: HKEY_LOCAL_MACHINE\Software\Python\PythonCore The installer is 32bit, so my OS shows it HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore when it requests HKEY_LOCAL_MACHINE\Software\Python\PythonCore The way to handle this would be either to get Python to register under both registry locations when installed on 64bit Windows (thus making itself visible to 32bit installers), or to make 64bit installers for the 64bit platform (so the OS would not try to trick them and push the wrong key). None of those sounds terribly complicated. Best regards, Andrey From david.lyon at preisshare.net Wed Dec 16 02:49:50 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 15 Dec 2009 20:49:50 -0500 Subject: [Distutils] =?utf-8?q?install=5Frequires_vs_requires?= In-Reply-To: <5b8d13220912142328ydb6823fg3c7bd26b2f005cc3@mail.gmail.com> References: <5b8d13220912142328ydb6823fg3c7bd26b2f005cc3@mail.gmail.com> Message-ID: <76bc3d77273fba58bc4464419acfddc5@preisshare.net> On Tue, 15 Dec 2009 12:58:16 +0530, David Cournapeau wrote: > Hi, > > I am working on a new packaging solution, and I want to convert > existing setup.py. Can I join in ? Is it a public project ? From cournape at gmail.com Wed Dec 16 08:19:50 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 16 Dec 2009 12:49:50 +0530 Subject: [Distutils] install_requires vs requires In-Reply-To: <94bdd2610912150110q64dbf5duc307e9a4e0d4139f@mail.gmail.com> References: <5b8d13220912142328ydb6823fg3c7bd26b2f005cc3@mail.gmail.com> <94bdd2610912150110q64dbf5duc307e9a4e0d4139f@mail.gmail.com> Message-ID: <5b8d13220912152319t5b30a15bvdfddb6b82af63411@mail.gmail.com> On Tue, Dec 15, 2009 at 2:40 PM, Tarek Ziad? wrote: > On Tue, Dec 15, 2009 at 8:28 AM, David Cournapeau wrote: >> Hi, >> >> I am working on a new packaging solution, and I want to convert >> existing setup.py. I noticed that the install_requires as used by >> setuptools is not put into PKG-INFO as requires. Is this expected ? > > Yes because it is not part of the standard, so setuptools is putting > this in "requires.txt" > which could be considered as an extension to the metadata. Ok, that makes sense. > > PEP 345 will include a new field that can replace requires.txt. > >> What is the difference between requires and require_install ? What >> happens when both have different values ? > > What are those names ? I mostly use the distutils sources as reference, and the DistributionMetadata class generates provides metadata as documented in the corresponding PEP if the requires attribute is not empty. Does it make sense to generate PKG-INFO with the requires field which value would be taken from the install_requires ? I am using sphinx as my main test for the conversion now, and I noticed that setuptools does not generate a PKG-INFO with those data (v1.0 instead of v1.1 as I would have expected). > See http://packages.python.org/distribute/setuptools.html#new-and-changed-setup-keywords > for the list of new keywords introduced by Setuptools (+ a few more by > Distribute) Will those arguments be added to distutils as well ? Or should I consider those arguments only when I detect distribute/setuptools-based setup.py ? cheers, David From tseaver at palladion.com Thu Dec 17 02:56:46 2009 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 16 Dec 2009 20:56:46 -0500 Subject: [Distutils] install_requires vs requires In-Reply-To: <5b8d13220912152319t5b30a15bvdfddb6b82af63411@mail.gmail.com> References: <5b8d13220912142328ydb6823fg3c7bd26b2f005cc3@mail.gmail.com> <94bdd2610912150110q64dbf5duc307e9a4e0d4139f@mail.gmail.com> <5b8d13220912152319t5b30a15bvdfddb6b82af63411@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Cournapeau wrote: > On Tue, Dec 15, 2009 at 2:40 PM, Tarek Ziad? wrote: >> On Tue, Dec 15, 2009 at 8:28 AM, David Cournapeau wrote: >>> Hi, >>> >>> I am working on a new packaging solution, and I want to convert >>> existing setup.py. I noticed that the install_requires as used by >>> setuptools is not put into PKG-INFO as requires. Is this expected ? >> Yes because it is not part of the standard, so setuptools is putting >> this in "requires.txt" >> which could be considered as an extension to the metadata. > > Ok, that makes sense. > >> PEP 345 will include a new field that can replace requires.txt. >> >>> What is the difference between requires and require_install ? What >>> happens when both have different values ? >> What are those names ? > > I mostly use the distutils sources as reference, and the > DistributionMetadata class generates provides metadata as documented > in the corresponding PEP if the requires attribute is not empty. > > Does it make sense to generate PKG-INFO with the requires field which > value would be taken from the install_requires ? I am using sphinx as > my main test for the conversion now, and I noticed that setuptools > does not generate a PKG-INFO with those data (v1.0 instead of v1.1 as > I would have expected). > >> See http://packages.python.org/distribute/setuptools.html#new-and-changed-setup-keywords >> for the list of new keywords introduced by Setuptools (+ a few more by >> Distribute) > > Will those arguments be added to distutils as well ? Or should I > consider those arguments only when I detect > distribute/setuptools-based setup.py ? The intent is to add the PEP 345 fields to PKG-INFO as generated by distutils. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkspj90ACgkQ+gerLs4ltQ7FowCgjauOk5CmewnCljm5W2qKdaBm YYoAnim9DoaUQ61Y+nJKAB+zalXfhoeW =ndG9 -----END PGP SIGNATURE----- From LuHe at gmx.at Thu Dec 17 08:51:04 2009 From: LuHe at gmx.at (Lukas Hetzenecker) Date: Thu, 17 Dec 2009 08:51:04 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <200912141649.42768.LuHe@gmx.at> References: <200912131508.36740.LuHe@gmx.at> <94bdd2610912140644t15b3e1fbg9f2a57d8185afc4b@mail.gmail.com> <200912141649.42768.LuHe@gmx.at> Message-ID: <200912170851.04694.LuHe@gmx.at> Hello, now I have only one problem left: If i call my setup.py script with the prefix argument (python setup.py install --prefix /home/lukas/tmp/) I can't get this argument in the script file. distutils.sysconfig.get_config_var("prefix") returns '/usr' and get_config_var("datarootdir") returns '/usr/share'. Is there any way to initialize the variables so that the command line arguments are noticed and that the function returns '/home/lukas/tmp/share'? Lukas > The layout for the release tarball should be like my development layout, > only the installation layout should be different. > > This means: The user downloads a tarball which has the package for his pc > (in the pc/ directory), a package for his mobile phone (in the mobile/ > directory) and the LICENCE, README files (in the root directory). > Now he wants to install the pc package. Only now the modules of the pc/ > directory are mixed with the README files and copied to the site-packages > directory. > > So he has the README files in the root directory of his tarball _and_ in > the package (site-packages/series60_remote) directory when he installed > the script. > > > What you have done will not work as-is because setup.py can be called > > for doing something else than 'sdist', so you need to run the code > > only if 'sdist' is called. > > Well, this "mixing" should _not_ happen when sdist is called. From cournape at gmail.com Thu Dec 17 10:19:05 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 17 Dec 2009 14:49:05 +0530 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <200912170851.04694.LuHe@gmx.at> References: <200912131508.36740.LuHe@gmx.at> <94bdd2610912140644t15b3e1fbg9f2a57d8185afc4b@mail.gmail.com> <200912141649.42768.LuHe@gmx.at> <200912170851.04694.LuHe@gmx.at> Message-ID: <5b8d13220912170119m18f6ac91t717ac1fca79545df@mail.gmail.com> On Thu, Dec 17, 2009 at 1:21 PM, Lukas Hetzenecker wrote: > Hello, > > now I have only one problem left: > If i call my setup.py script with the prefix argument (python setup.py install > --prefix /home/lukas/tmp/) I can't get this argument in the script file. > > distutils.sysconfig.get_config_var("prefix") returns '/usr' and > get_config_var("datarootdir") returns '/usr/share'. Is there any way to > initialize the variables so that the command line arguments are noticed and > that the function returns '/home/lukas/tmp/share'? No, at least not using public API. What you can do is to override the install command (the one which knows the computed install prefix), and use it from there. But you still cannot control this from the setup.py, that is your command is called by distutils, and you don't control it. David From eemeli.kantola at iki.fi Thu Dec 17 17:10:26 2009 From: eemeli.kantola at iki.fi (Eemeli Kantola) Date: Thu, 17 Dec 2009 18:10:26 +0200 Subject: [Distutils] Install egg from SVN sources => AttributeError: 'module' object has no attribute '__getstate__' Message-ID: <48832f000912170810p14f702afjc14b4bd62d19eb31@mail.gmail.com> Hi all, I'm getting a crash in setuptools-0.6c11-py2.6 when trying to install an egg directly from SVN sources. This happens both on OSX and Ubuntu, versions: Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) [GCC 4.3.3] on linux2 Command and stack trace: $ sudo easy_install https://asibsync.svn.sourceforge.net/svnroot/asibsync/src/asilib Downloading https://asibsync.svn.sourceforge.net/svnroot/asibsync/src/asilib Doing subversion checkout from https://asibsync.svn.sourceforge.net/svnroot/asibsync/src/asilib to /tmp/easy_install-WYCRs6/asilib Processing asilib Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-WYCRs6/asilib/egg-dist-tmp-6_PSoY Traceback (most recent call last): File "/usr/local/bin/easy_install", line 8, in load_entry_point('setuptools==0.6c11', 'console_scripts', 'easy_install')() File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 1712, in main File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 1716, in File "/usr/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 211, in run File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 422, in easy_install File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 476, in install_item File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 655, in install_eggs File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 930, in build_and_install File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", line 919, in run_setup File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/sandbox.py", line 52, in run_setup AttributeError: 'module' object has no attribute '__getstate__' This is the first time I look at setuptools' code, but it seems the problem goes away when I conditionally disabled the __getstate__ and __setstate__ lines in sandbox.py, as per this batch: --- sandbox.py.old 2009-10-19 13:35:44.000000000 +0300 +++ sandbox.py 2009-12-17 17:49:10.000000000 +0200 @@ -49,7 +49,8 @@ if not os.path.isdir(temp_dir): os.makedirs(temp_dir) save_tmp = tempfile.tempdir save_modules = sys.modules.copy() - pr_state = pkg_resources.__getstate__() + if hasattr(pkg_resources, '__getstate__'): + pr_state = pkg_resources.__getstate__() try: tempfile.tempdir = temp_dir; os.chdir(setup_dir) try: @@ -69,7 +70,8 @@ raise # Normal exit, just return finally: - pkg_resources.__setstate__(pr_state) + if hasattr(pkg_resources, '__getstate__'): + pkg_resources.__setstate__(pr_state) sys.modules.update(save_modules) for key in list(sys.modules): if key not in save_modules: del sys.modules[key] So is this a bug? My fix is presumably only treating the symptoms, because I've got only little clue of what is happening in the code... Best, Eemeli Kantola From pje at telecommunity.com Fri Dec 18 04:24:15 2009 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 17 Dec 2009 22:24:15 -0500 Subject: [Distutils] Install egg from SVN sources => AttributeError: 'module' object has no attribute '__getstate__' In-Reply-To: <48832f000912170810p14f702afjc14b4bd62d19eb31@mail.gmail.co m> References: <48832f000912170810p14f702afjc14b4bd62d19eb31@mail.gmail.com> Message-ID: <20091218032435.581A23A408B@sparrow.telecommunity.com> At 06:10 PM 12/17/2009 +0200, Eemeli Kantola wrote: >Hi all, > >I'm getting a crash in setuptools-0.6c11-py2.6 when trying to install >an egg directly from SVN sources. This happens both on OSX and Ubuntu, >versions: > >Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) [GCC 4.2.1 (Apple >Inc. build 5646)] on darwin >Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) [GCC 4.3.3] on linux2 > >Command and stack trace: >... > >So is this a bug? My fix is presumably only treating the symptoms, >because I've got only little clue of what is happening in the code... My guess is that you've got some other pkg_resources module on your sys.path, such as from an older version of setuptools. Could you check the following from your Python prompt? >>> import pkg_resources, setuptools >>> print pkg_resources.__file__ >>> print setuptools.__file__ >>> print setuptools.__version__ Thanks. From devyan.parmar at gmail.com Fri Dec 18 13:33:54 2009 From: devyan.parmar at gmail.com (devyan parmar) Date: Fri, 18 Dec 2009 18:03:54 +0530 Subject: [Distutils] Not picking Older version eggs Message-ID: Hi, I am using buildout for my project development as well as for deployment. which makes process easy and help in repeatability. but facing one problem while choosing particular versions of eggs while running buildout. i have eggified my python packages and releasing version wise. i have created "versions.cfg" where i mentioned all my project related packages with proper version. it's working fine for latest eggs version. but if i wants to choose older version egg from my package server, It takes latest only. even after specifying in versions.cfg also. following attributes am using in my buildout.cfg [buildout] newest = false prefer-final = true allow-picked-versions = true extends = version.cfg preferences.cfg versions = versions to avoid this i am deleting existing packages from my eggs folder and rerunning buildout. Please suggest me here... Thanks Regards Devyan Parmar -------------- next part -------------- An HTML attachment was scrubbed... URL: From eemeli.kantola at iki.fi Fri Dec 18 14:23:47 2009 From: eemeli.kantola at iki.fi (Eemeli Kantola) Date: Fri, 18 Dec 2009 15:23:47 +0200 Subject: [Distutils] Install egg from SVN sources => AttributeError: 'module' object has no attribute '__getstate__' In-Reply-To: <20091218032435.581A23A408B@sparrow.telecommunity.com> References: <48832f000912170810p14f702afjc14b4bd62d19eb31@mail.gmail.com> <20091218032435.581A23A408B@sparrow.telecommunity.com> Message-ID: <48832f000912180523p52902c5ci5f90b3cba6f254ff@mail.gmail.com> 2009/12/18 P.J. Eby : > At 06:10 PM 12/17/2009 +0200, Eemeli Kantola wrote: >> >> Hi all, >> >> I'm getting a crash in setuptools-0.6c11-py2.6 when trying to install >> an egg directly from SVN sources. This happens both on OSX and Ubuntu, >> versions: >> >> Python 2.6.1 (r261:67515, Jul ?7 2009, 23:51:51) [GCC 4.2.1 (Apple >> Inc. build 5646)] on darwin >> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) [GCC 4.3.3] on >> linux2 >> >> Command and stack trace: >> ... >> >> So is this a bug? My fix is presumably only treating the symptoms, >> because I've got only little clue of what is happening in the code... > > My guess is that you've got some other pkg_resources module on your > sys.path, such as from an older version of setuptools. > > Could you check the following from your Python prompt? > >>>> import pkg_resources, setuptools >>>> print pkg_resources.__file__ >>>> print setuptools.__file__ >>>> print setuptools.__version__ Yes, thank you! Probably that was it, because after reinstalling setuptools 0.6c11, the problem went magically away. On OSX, setuptools was clearly somehow broken, but on Ubuntu side, there seemed to be no obvious problems before reinstalling: >>> import pkg_resources, setuptools >>> print pkg_resources.__file__ /usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py >>> print setuptools.__file__ /usr/local/lib/python2.6/dist-packages/setuptools-0.6c11-py2.6.egg/setuptools/__init__.py >>> print setuptools.__version__ 0.6c11 Anyway, problem solved. > Thanks. Best, -Eemeli From ziade.tarek at gmail.com Fri Dec 18 15:41:48 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 18 Dec 2009 15:41:48 +0100 Subject: [Distutils] Finishing PEP 345 Message-ID: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> Hi, While PEP 386 is waiting to be accepted in python-dev, I'd like to finish PEP 345 so we can propose it right after PEP 386 is (hopefully) accepted. The remaining point I can see is about : Repository-URL, Repository-Browse-URL, Bug-Tracker-URL. As Ian suggested I propose to change these three fields to: Project-URL (multiple-use) : A string containing an URL for the project and a label for it. separated by a comma Example:: Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ This field, like Requires-Dist, can be present multiple times. The label of the URL is preferrably unique but there will be no control of this unicity. Although the label is limited to 32 signs so the developer stay concise. The URL is a valid browsable URL. (http/ftp/https). PyPI could then use these URLs on the project page like this: Bug Tracker Any thoughts ? Tarek -- Tarek Ziad? | http://ziade.org From dsdale24 at gmail.com Fri Dec 18 15:56:28 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Fri, 18 Dec 2009 09:56:28 -0500 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> Message-ID: On Fri, Dec 18, 2009 at 9:41 AM, Tarek Ziad? wrote: > Hi, > > While PEP 386 is waiting to be accepted in python-dev, I'd like to > finish PEP 345 so we can propose it right after PEP 386 is (hopefully) > accepted. > > The remaining point I can see is about : Repository-URL, > Repository-Browse-URL, Bug-Tracker-URL. > > As Ian suggested I propose to change these three fields to: > > Project-URL (multiple-use) : > > ? ?A string containing an URL for the project and a label for it. > separated by a comma > > ? ?Example:: > > ? ? ? ? ? Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ > > > This field, like Requires-Dist, can be present multiple times. The > label of the URL is preferrably unique > but there will be no control of this unicity. Although the label is > limited to 32 signs so the developer stay concise. > The URL is a valid browsable URL. (http/ftp/https). > > PyPI could then use these URLs on the project page like this: > > ?Bug Tracker > > > Any thoughts ? I have a question about fields being present multiple times. Should the format be compatible with ConfigParser? I don't think ConfigParser knows how to handle this situation. Darren From dsdale24 at gmail.com Fri Dec 18 16:08:58 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Fri, 18 Dec 2009 10:08:58 -0500 Subject: [Distutils] Finishing PEP 345 In-Reply-To: References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> Message-ID: On Fri, Dec 18, 2009 at 9:56 AM, Darren Dale wrote: > On Fri, Dec 18, 2009 at 9:41 AM, Tarek Ziad? wrote: >> Hi, >> >> While PEP 386 is waiting to be accepted in python-dev, I'd like to >> finish PEP 345 so we can propose it right after PEP 386 is (hopefully) >> accepted. >> >> The remaining point I can see is about : Repository-URL, >> Repository-Browse-URL, Bug-Tracker-URL. >> >> As Ian suggested I propose to change these three fields to: >> >> Project-URL (multiple-use) : >> >> ? ?A string containing an URL for the project and a label for it. >> separated by a comma >> >> ? ?Example:: >> >> ? ? ? ? ? Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ >> >> >> This field, like Requires-Dist, can be present multiple times. The >> label of the URL is preferrably unique >> but there will be no control of this unicity. Although the label is >> limited to 32 signs so the developer stay concise. >> The URL is a valid browsable URL. (http/ftp/https). >> >> PyPI could then use these URLs on the project page like this: >> >> ?Bug Tracker >> >> >> Any thoughts ? > > I have a question about fields being present multiple times. Should > the format be compatible with ConfigParser? I don't think ConfigParser > knows how to handle this situation. Sorry, I think I confused this with declaring metadata in setup.cfg (PEP 390). Darren From aljosa.mohorovic at gmail.com Fri Dec 18 18:12:10 2009 From: aljosa.mohorovic at gmail.com (=?UTF-8?B?QWxqb8WhYSBNb2hvcm92acSH?=) Date: Fri, 18 Dec 2009 18:12:10 +0100 Subject: [Distutils] install_requires option to force pypi url or a better approach? Message-ID: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.com> i've setup private pypi where i can upload some packages that are not suitable for pypi.python.org. my package, let's call it myapp, contains "install_requires=['django',]" but with that requirement i can't install it. -------------------------------------------------- $ virtualenv --no-site-packages --distribute env New python executable in env/bin/python A globally installed setuptools was found (in /usr/lib/python2.6/dist-packages) Use the --no-site-packages option to use distribute in the virtualenv. Installing distribute...........................................................................................................................................................................done. $ . env/bin/activate $ pip freeze distribute==0.6.8 wsgiref==0.1.2 $ pip install myapp -i http://pypi.example.com Downloading/unpacking myapp ?Using download cache from /home/aljosa/.pip-cache/ ?Running setup.py egg_info for package myapp Downloading/unpacking django (from myapp) ?Could not find any downloads that satisfy the requirement django (from myapp) No distributions at all found for django (from myapp) -------------------------------------------------- since i can normally install django with "pip install django" i assume that when i set "-i http://pypi.example.com" it's also used for install_requires. any tips on using multiple pypi repositories? is there some way to define which pypi url to use for package installation? Aljosa Mohorovic From ziade.tarek at gmail.com Fri Dec 18 18:48:57 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 18 Dec 2009 18:48:57 +0100 Subject: [Distutils] install_requires option to force pypi url or a better approach? In-Reply-To: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.com> References: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.com> Message-ID: <94bdd2610912180948i3dea0ef8rd1976b8be77a6e21@mail.gmail.com> On Fri, Dec 18, 2009 at 6:12 PM, Aljo?a Mohorovi? wrote: [..] > $ pip freeze > distribute==0.6.8 > wsgiref==0.1.2 > $ pip install myapp -i http://pypi.example.com [..] Have you tried --extra-index-url ? This allows you to add extra indexes Regards Tarek -- Tarek Ziad? | http://ziade.org From aljosa.mohorovic at gmail.com Fri Dec 18 18:59:45 2009 From: aljosa.mohorovic at gmail.com (=?UTF-8?B?QWxqb8WhYSBNb2hvcm92acSH?=) Date: Fri, 18 Dec 2009 18:59:45 +0100 Subject: [Distutils] install_requires option to force pypi url or a better approach? In-Reply-To: <94bdd2610912180948i3dea0ef8rd1976b8be77a6e21@mail.gmail.com> References: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.com> <94bdd2610912180948i3dea0ef8rd1976b8be77a6e21@mail.gmail.com> Message-ID: <87d364ab0912180959t497d215bt392455716807e596@mail.gmail.com> 2009/12/18 Tarek Ziad? : > Have you tried --extra-index-url ? This allows you to add extra indexes i can define pypi.example.com in requirements.txt or via command line options for pip but i'm trying to define it in setup.py. is it unreasonable to expect option in setup.py to define multiple/alternative pypi locations or should i change something in my workflow? telling pip where to look for each package requirement (assuming it's in a different pypi location) makes me think that i shouldn't define requirements in setup.py and simply use pip/requirements.txt to manage dependencies. maybe i'm missing something? Aljosa From ziade.tarek at gmail.com Fri Dec 18 19:14:20 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 18 Dec 2009 19:14:20 +0100 Subject: [Distutils] install_requires option to force pypi url or a better approach? In-Reply-To: <87d364ab0912180959t497d215bt392455716807e596@mail.gmail.com> References: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.com> <94bdd2610912180948i3dea0ef8rd1976b8be77a6e21@mail.gmail.com> <87d364ab0912180959t497d215bt392455716807e596@mail.gmail.com> Message-ID: <94bdd2610912181014w6c9f2f82gd9f582ba430675dd@mail.gmail.com> 2009/12/18 Aljo?a Mohorovi? : > 2009/12/18 Tarek Ziad? : >> Have you tried --extra-index-url ? This allows you to add extra indexes > > i can define pypi.example.com in requirements.txt or via command line > options for pip but i'm trying to define it in setup.py. > is it unreasonable to expect option in setup.py to define > multiple/alternative pypi locations or should i change something in my > workflow? > telling pip where to look for each package requirement (assuming it's > in a different pypi location) makes me think that i shouldn't define > requirements in setup.py and simply use pip/requirements.txt to manage > dependencies. > > maybe i'm missing something? > Usually a setup.py script just provides the options for the install command to be run, not extra options for a package installer like Pip. IOW, defining where the package installer should look for distributions should be configured at its level, not at the distribution level, because one distribution may be located in several PyPI-like servers and cannot know in advance which installer will be used for it to be installed, and on which server it will be hosted at So for Pip, imho you should list all your pypi servers using --extra-index-url and leave the distributions' setup.py alone. Tarek From aljosa.mohorovic at gmail.com Fri Dec 18 19:23:15 2009 From: aljosa.mohorovic at gmail.com (=?UTF-8?B?QWxqb8WhYSBNb2hvcm92acSH?=) Date: Fri, 18 Dec 2009 19:23:15 +0100 Subject: [Distutils] install_requires option to force pypi url or a better approach? In-Reply-To: <94bdd2610912181014w6c9f2f82gd9f582ba430675dd@mail.gmail.com> References: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.com> <94bdd2610912180948i3dea0ef8rd1976b8be77a6e21@mail.gmail.com> <87d364ab0912180959t497d215bt392455716807e596@mail.gmail.com> <94bdd2610912181014w6c9f2f82gd9f582ba430675dd@mail.gmail.com> Message-ID: <87d364ab0912181023s32632cbes9351c4b7e8b99c62@mail.gmail.com> Makes sense, I'll try to adapt. Thanks for info On Dec 18, 2009 7:14 PM, "Tarek Ziad?" wrote: 2009/12/18 Aljo?a Mohorovi? : > 2009/12/18 Tarek Ziad? : >> Have you tried --extra-index-url ? This allows ... Usually a setup.py script just provides the options for the install command to be run, not extra options for a package installer like Pip. IOW, defining where the package installer should look for distributions should be configured at its level, not at the distribution level, because one distribution may be located in several PyPI-like servers and cannot know in advance which installer will be used for it to be installed, and on which server it will be hosted at So for Pip, imho you should list all your pypi servers using --extra-index-url and leave the distributions' setup.py alone. Tarek -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Fri Dec 18 23:58:40 2009 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 18 Dec 2009 17:58:40 -0500 Subject: [Distutils] install_requires option to force pypi url or a better approach? In-Reply-To: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.co m> References: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.com> Message-ID: <20091218225903.550583A408B@sparrow.telecommunity.com> At 06:12 PM 12/18/2009 +0100, Aljo?a Mohorovi?? wrote: >i've setup private pypi where i can upload some packages that are >not suitable for pypi.python.org. my package, let's call it myapp, >contains "install_requires=['django',]" but with that requirement i >can't install it. -------------------------------------------------- >$ virtualenv --no-site-packages --distribute env New python >executable in env/bin/python A globally installed setuptools was >found (in /usr/lib/python2.6/dist-packages) Use the >--no-site-packages option to use distribute in the virtualenv. >Installing >distribute...........................................................................................................................................................................done. >$ . env/bin/activate $ pip freeze distribute==0.6.8 wsgiref==0.1.2 $ >pip install myapp -i http://pypi.example.com Downloading/unpacking >myapp ? Using download cache from /home/aljosa/.pip-cache/URL> ? Running setup.py egg_info for package myapp >Downloading/unpacking django (from myapp) ? Could not find any >downloads that satisfy the requirement django (from myapp) No >distributions at all found for django (from myapp) >-------------------------------------------------- since i can >normally install django with "pip install django" i assume that when >i set "-i http://pypi.example.com" it's also used for >install_requires. any tips on using multiple pypi repositories? is >there some way to define which pypi url to use for package installation? You would probably be better off using the find_links option in your setup(), to indicate additional URLs where packages can be found. e.g. find_links=['http://pypi.example.com/somepackage', 'http://pypi.example.com/otherpackage']. This will tell easy_install (or other packaging tools) to check these pages in addition to the standard package-index pages, when seeking out your dependencies. From ziade.tarek at gmail.com Sat Dec 19 00:37:21 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 19 Dec 2009 00:37:21 +0100 Subject: [Distutils] install_requires option to force pypi url or a better approach? In-Reply-To: <20091218225903.550583A408B@sparrow.telecommunity.com> References: <87d364ab0912180912i46187eb8y5b69268f4f609bd1@mail.gmail.com> <20091218225903.550583A408B@sparrow.telecommunity.com> Message-ID: <94bdd2610912181537p36c00dd3j8fc6e7dfceb01aad@mail.gmail.com> On Fri, Dec 18, 2009 at 11:58 PM, P.J. Eby wrote: [..] > > You would probably be better off using the find_links option in your > setup(), to indicate additional URLs where packages can be found. ?e.g. > find_links=['http://pypi.example.com/somepackage', > 'http://pypi.example.com/otherpackage']. ?This will tell easy_install (or > other packaging tools) to check these pages in addition to the standard > package-index pages, when seeking out your dependencies. I think it's pretty tedious to add a find_links entry for each dependency, when they are all available on some PyPI servers. Since Pip allows to configure two PyPI servers at installation time, And if the distribution is released in some other repository some day (let's say PyPI) and these urls are internal, this list will have to be changed accordingly. >From my experience, adding some find-links in a distribution's setup.py is a bad practice in general: for instance, if you want to change the package index in the package installer to control where files are downloaded, find-links will by-pass this unless you use the "allow-hosts" option. I had trouble with this problem for example in some places where people had a firewall : if *one* package had a find-links, the installer was trying to look it up, even if I had a full local PyPI mirror So I had to use the allow-host option to avoid this. And just because some dependency I didn't control had a find-link. If some dependencies are not available at PyPI. the way to get them and install them should be documented but not forced imho. And in Aljo?a's case, they are in a PyPI-like server, so it's just a matter of configuring pip. Tarek From setuptools at bugs.python.org Sat Dec 19 07:25:08 2009 From: setuptools at bugs.python.org (Phil Mitchell) Date: Sat, 19 Dec 2009 06:25:08 +0000 Subject: [Distutils] [issue99] easy_install does not honour exec-prefix for platform-dependent eggs In-Reply-To: <1261203908.31.0.877685886428.issue99@psf.upfronthosting.co.za> Message-ID: <1261203908.31.0.877685886428.issue99@psf.upfronthosting.co.za> New submission from Phil Mitchell : When running easy_install with a platform-dependent egg on a python installation whose exec-prefix differs from its prefix, the egg is installed in the platform-independent site-packages directory. In some senses, this seems okay because the egg name contains the platform name, but the easy_install.pth file only includes the egg from one platform, and if multiple eggs with the same name (but different platform) are added to the easy_install.pth file, then only the first will be considered. This comes up when using a common installation of python on a network share with different exec-prefix settings for each platform. This would not be an issue if easy_install were to install platform-dependent eggs into the platform-specific site-packages location and set up the easy_install.pth for that location appropriately. I am able to set things up this way manually, but it is not quite as "easy" to "install." ---------- messages: 477 nosy: philbert priority: bug status: unread title: easy_install does not honour exec-prefix for platform-dependent eggs _______________________________________________ Setuptools tracker _______________________________________________ From ssteinerx at gmail.com Sat Dec 19 18:33:40 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sat, 19 Dec 2009 12:33:40 -0500 Subject: [Distutils] Distribute testing Message-ID: <5F3DDD8E-DB38-4685-AAA5-E5E69D8BD1EE@gmail.com> Tarek, We recently talked about incorporating the testing scheme I started in distutils-buildbot directly into the release path for Distutils to avoid issues like the ones with the most recent release. I am putting this out on the list for comments; I'm NOT interested in bikeshedding this to death. Anyone else is welcome to do their own project if they feel strongly enough about it. I'm going to take opinions into account but I want to have the tool built before the next release, not the next millennium release of Python. The way I started with the project in distutils-buildbot (http://bitbucket.org/tarek/distutils-buildbot/ smoke-and-mirrors branch, bin/smoke2.py, bin/smoke2.cfg, and bin/smoke_base_products.cfg) was to assume a brand new system e.g. a newly initialized Cloud Server and to build out everything necessary to run the tests under Python 2.4, 2.5, 2.6, 2.7dev, 3.1, and 3.2dev. While this approach will be fine when we move it to building on-demand buildbots, it is much too time consuming both in development time and real-time processing to use for this release cycle. It would be much better (for now) to just make sure you're running it on a machine that can do the job. The main goal is to come up with a simple way to run as many full-scope tests i.e. from install through full test-suite on as many large products, with as many versions of Python as practical. What we had originally started was to build and run the test suites for (spec'd in bin/smoke_base_products.cfg): Twisted numpy tarek_extensions warped I had stopped where it reads the configuration files and knows on what versions of Python to test which products. At this point, I'd like to move this into Distribute proper and get it into the "pre-release" test process. So... First, we need to decide where to put the testing sub-project. I'm asuming it can just live in the /tests subdirectory, maybe with a subdir to hold this whole project as its own module. Here's what I'd like to see it do: Run pre-test checks i.e. verify that the machine is capable of running the tests: 1> Assume that `python2.4`, `python2.5`, `python2.6`, `python3.1` will invoke a properly set up Python interpreter. This is beyond the scope of this project; might be taken up at a later time. 2> Create a simple way to verify installed utilities before proceeding with tests (anyone have one of these around?). Something like: ASSERT(`nosetests -V` == `nosetests version 0.11.1`) 3> Allow a certain amount of corrective action to be able to be taken to satisfy the assertions above (like, install nose version 0.1.11 from PyPi, for example) for each supported version of Python. Then: for each product to be tested: for each Python version this product is to be tested on: install the product run its test suite record any failures Pretty simple, but will avoid any of the sorts of issues we just had with the most recent release, is very easy to add new products to (just add a configuration section), and can be easily moved out to the buildbots when it does what we want. Comments etc. gladly accepted as long as there are no bikeshedding tools (hammers, paint-brushes, Erlang) in hand. Thanks, S aka/Steve Steiner aka/ssteinerX From floris.bruynooghe at gmail.com Sat Dec 19 22:12:28 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sat, 19 Dec 2009 21:12:28 +0000 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> Message-ID: <20091219211228.GA10616@laurie.devork> On Fri, Dec 18, 2009 at 03:41:48PM +0100, Tarek Ziad? wrote: > The remaining point I can see is about : Repository-URL, > Repository-Browse-URL, Bug-Tracker-URL. > > As Ian suggested I propose to change these three fields to: > > Project-URL (multiple-use) : > > A string containing an URL for the project and a label for it. > separated by a comma [...] > The URL is a valid browsable URL. (http/ftp/https). How about dropping the word "browsable" from there as well as the requirement on the schema? I like the complete free-form approach to this tough. But the downside is that it will be harder for indexes like PyPI to include relvant links in their UI. There will have to be some sort of "agreed" string for a bug tracker for example. How forceful are you going to be with the 32 character limit? Reject it or just print warnings? Regards Floris From ziade.tarek at gmail.com Sat Dec 19 23:05:57 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 19 Dec 2009 23:05:57 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <20091219211228.GA10616@laurie.devork> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> Message-ID: <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> On Sat, Dec 19, 2009 at 10:12 PM, Floris Bruynooghe wrote: > On Fri, Dec 18, 2009 at 03:41:48PM +0100, Tarek Ziad? wrote: >> The remaining point I can see is about : Repository-URL, >> Repository-Browse-URL, Bug-Tracker-URL. >> >> As Ian suggested I propose to change these three fields to: >> >> Project-URL (multiple-use) : >> >> ? ? A string containing an URL for the project and a label for it. >> separated by a comma > [...] >> The URL is a valid browsable URL. (http/ftp/https). > > How about dropping the word "browsable" from there as well as the > requirement on the schema? ?I like the complete free-form approach to > this tough. > > But the downside is that it will be harder for indexes like PyPI to > include relvant links in their UI. ?There will have to be some sort of > "agreed" string for a bug tracker for example. Yes that's the point: making links that can be clicked by users. What kind of other consumers do you have in mind ? Since it's a free list, I can't imagine another consumer than some kind of UI that displays them. > > How forceful are you going to be with the 32 character limit? ?Reject > it or just print warnings? truncate them in distutils when sdist is called, with a warning. 32 characters allows most meaningful labels: "Bug tracker" "Repository" "Documentation" "Mailing List" But the PKG-INFO will not contain more than 3 characters once it is built Regards Tarek From ziade.tarek at gmail.com Sun Dec 20 12:54:14 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 20 Dec 2009 12:54:14 +0100 Subject: [Distutils] Distribute testing In-Reply-To: <5F3DDD8E-DB38-4685-AAA5-E5E69D8BD1EE@gmail.com> References: <5F3DDD8E-DB38-4685-AAA5-E5E69D8BD1EE@gmail.com> Message-ID: <94bdd2610912200354j378407fbm63f7bb0a1b58e450@mail.gmail.com> On Sat, Dec 19, 2009 at 6:33 PM, ssteinerX at gmail.com wrote: [..] > > ? ? ? ?While this approach will be fine when we move it to building on-demand buildbots, it is much too time consuming both in development time and real-time processing to use for this release cycle. ?It would be much better (for now) to just make sure you're running it on a machine that can do the job. *or* to update an existing test environment. e.g. check if python != trunk is already present. [..] > ? ? ? ?First, we need to decide where to put the testing sub-project. ?I'm asuming it can just live in the /tests subdirectory, maybe with a subdir to hold this whole project as its own module. Sounds good > > ? ? ? ?Here's what I'd like to see it do: > > ? ? ? ?Run pre-test checks i.e. verify that the machine is capable of running the tests: > > ? ? ? ? ? ? ? ?1> ? ? ?Assume that `python2.4`, `python2.5`, `python2.6`, `python3.1` will invoke a properly ? set up Python interpreter. ?This is beyond the scope of this project; might be taken up at a later time. > > ? ? ? ? ? ? ? ?2> ? ? ?Create a simple way to verify installed utilities before proceeding with tests (anyone have one of these around?). > ? ? ? ? ? ? ? ?Something like: > ? ? ? ? ? ? ? ? ? ? ? ?ASSERT(`nosetests -V` == `nosetests version 0.11.1`) > > ? ? ? ? ? ? ? ?3> ? ? ?Allow a certain amount of corrective action to be able to be taken to satisfy the assertions above (like, install nose version 0.1.11 from PyPi, for example) for each supported version of Python. > > ? ? ? ?Then: > ? ? ? ? ? ? ? ?for each product to be tested: > ? ? ? ? ? ? ? ? ? ? ? ?for each Python version this product is to be tested on: > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?install the product > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?run its test suite > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?record any failures > > ? ? ? ?Pretty simple, but will avoid any of the sorts of issues we just had with the most recent > release, is very easy to add new products to (just add a configuration section), and can be easily > moved out to the buildbots when it does what we want. Sounds good too ! Notice that we also need to try those tests in various environments, besides python versions: virtualenv, no virtualenv, setuptools present before we start, not present, etc Regards, Tarek From smueller at cpan.org Sun Dec 20 16:21:04 2009 From: smueller at cpan.org (Steffen Mueller) Date: Sun, 20 Dec 2009 15:21:04 +0000 (UTC) Subject: [Distutils] Python people want CPAN and how the latter came about Message-ID: It'll be no secret to anyone reading this that there has been a lot of discussion recently about a CPAN equivalent for Python, sparked by Guido's "People want CPAN" post to the python-distutils-devel list. I read a good chunk of the thread back in November and have been meaning to add my blurb since. Take it with a grain of salt, though. My knowledge of the language is weak, my knowledge of the community is weaker. I'm from the Perl crowd[1] and would simply like to share my late comer's view of how the CPAN came about. My thesis is that the huge success of the CPAN has been facilitated by two factors[2]. The first is simplicity. When Jarkko Hietaniemi originally came up with it, the CPAN was (and mostly still is) just an FTP archive with a by-author directory structure that is mirrored many times. Everything else is built on this flexible foundation and has grown over time. The CPAN specifically does NOT have an official web service or any kind of development platform. Apart from the directory structure of the CPAN, the only other key ingredient was the "Perl Authors Upload SErver" (PAUSE) that handles credentials of the authors, permissions for namespaces, and serves as a single entry-point for uploads to the CPAN. PAUSE scans incoming distributions for meta information and generates in index of modules (namespaces/classes) and distributions that is itself distributed via the simple CPAN mechanism. Let me repeat: Everything else is just sugar on top. Specifically, everything else is sugar provided by *third parties*. Andreas K?nig wrote and still maintains PAUSE and the CPAN.pm client[3]. Later, Jos Boumans set out to write the CPANPLUS client. Graham Barr wrote and still maintains the search.cpan.org website. Randy Kobes wrote the similar but equally non-official kobesearch.cpan.org. Many other significant pieces of the infrastructure[4] have been written and are maintained by other people who did cooperate with each other but never had to. By virtue of the simple design and the (somewhat limited) published meta-data in form of the simple module index, everyone had the opportunity to write tools that interface with it. There was no need to "get it all right" from the get-go. Things evolved and we now have a best-of-breed. The various services by various people are loosely intertwined[5]. There is also very little regulation on what is uploaded to CPAN, but curiously little abuse. I think that is because the majority of people who are willing to share their work with others free of charge aren't the type who'd want to crap on other people's front yard. If I was to set up something like CPAN today, I'd put in somewhat more design, simply because we have learned from the past fifteen years of operation. I'd specifically put in some more work on namespace and distribution-level meta data[6]. But I'd make very sure to keep it all relatively simple. By no means would I want to run any sort of elaborate web service beyond the authentication and authorization that the PAUSE provides. This eliminates some of the reliance on a very, very strong single party to provide initial implementation, hosting and maintenance for what is a very significant piece of software. My firm belief is that the second most important factor to the success of the CPAN is people. There are some individuals who have managed herculean amounts of work and have shown incredible dedication over years. But it's the combination of a lot of people's work that is more than its sum. I'd say the core of the CPAN toolchain gang is not that large. Depending on where you draw the line, there are maybe 10-50 of them. Not all, but many of these people have known each other personally for years from attending the many YAPCs and Perl workshops. Reading the "People want CPAN" thread, it seemed to me that folks are fighting each other quite a bit and not always only on technical grounds. On the other hand, it's hard to imagine fighting with somebody as friendly and welcoming as, for example, Andreas K?nig. Meeting in person helps this tremendously. Discover the common goals and agree on the means over a beer. I'm certain that is happening in the Python world. But looking at the prices of, say, attending PyCon, I wonder whether such an event can have the same spirit and community building effect as a YAPC or even the smaller workshops[7]. So if anyone was to accept a single concrete suggestion from me, it'd be: "Enable the right heads to get together over a beer." It is the bonds between individuals that can make it all work well. The success of the CPAN is due to cultural aspects at least as much as due to its design and technical merits. Please keep in mind that this is only my personal opinion. Thanks for reading. I hope this can add some perspective. Best regards, Steffen Mueller PS: Please keep me in CC of relevant discussion. I'm not subscribed to this list. [1] So why do I think I have anything to contribute to this discussion? I'm a regular contributor to the CPAN and a very modest contributor to the perl core distribution. I maintain over a hundred Perl modules, am one of the bunch of PAUSE admins who try to keep the chaos sane, and have been involved in Perl & CPAN toolchain maintenance to some degree. [2] You could argue that having a CTAN to take inspiration from helped, too, but I'm too young to know the exact stage of development of the CTAN in 1995. [3] Recently, the tireless David Golden has been doing a lot maintenance, too. [4] As another example, I'd like to point out the CPAN testers as one of the most important bits of the CPAN infrastructure. The setup is as simple as that of CPAN itself, but to some degree, it hasn't aged as well. It its core, the CPAN testers is just a mailing list to which anybody can send test reports. This is done by volunteers who spend their and their computers' time on testing CPAN distributions, but anybody can easily set up their CPAN client to send test reports while installing modules. The wealth of reports (I think we hit 6 million recently) has become a strain on the software that runs the mailing list archive. People are working on modernization of the setup. The example demonstrates how decentralization works well. There is no *official* cluster of computers that automatically tests new submissions. It's all volunteers who test on their hardware (usually automatically). [5] Does that ring a bell? Sounds a little "web 2.0" with less polish to me. From fifteen years ago. [6] There is a working group for better meta information in CPAN distributions, initiated and carried forward by David Golden. Most of the discussion is done. There is a consensus on a large fraction of the proposals. A draft specification is forthcoming. Implementation in the CPAN toolchain will likely follow the draft shortly and will be part of perl 5 release 14. (Release 12 is already feature-frozen.) [7] It's not that long ago that I was a student. $225 student rate for attending a conference? Wouldn't have been able to afford it. The standard individual rate is also ~3.5 times as high for PyCon as it is for YAPC. I think that's a harmful barrier to entry. From ssteinerx at gmail.com Sun Dec 20 17:15:19 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 20 Dec 2009 11:15:19 -0500 Subject: [Distutils] Distribute testing In-Reply-To: <94bdd2610912200354j378407fbm63f7bb0a1b58e450@mail.gmail.com> References: <5F3DDD8E-DB38-4685-AAA5-E5E69D8BD1EE@gmail.com> <94bdd2610912200354j378407fbm63f7bb0a1b58e450@mail.gmail.com> Message-ID: <39B1D5D4-09D8-4027-AD68-8A025A33F1F6@gmail.com> On Dec 20, 2009, at 6:54 AM, Tarek Ziad? wrote: > On Sat, Dec 19, 2009 at 6:33 PM, ssteinerX at gmail.com > wrote: > [..] >> >> While this approach will be fine when we move it to building on-demand buildbots, it is much too time consuming both in development time and real-time processing to use for this release cycle. It would be much better (for now) to just make sure you're running it on a machine that can do the job. > > *or* to update an existing test environment. e.g. check if python != trunk is already present. Yes, but what I mean is that I'm not going to be downloading and building all supported versions of Python itself in this version as I will in the buildbot version. Updating the dev versions of 2.7 and 3.2 to latest trunk will be in this version as will updating any peripheral tools required for testing (pip etc.). >> First, we need to decide where to put the testing sub-project. I'm asuming it can just live in the /tests subdirectory, maybe with a subdir to hold this whole project as its own module. > > Sounds good Ok, I'll do that. I'm going to put this in one working chunk at a time so that you can use whatever's there right away. As time goes on, more tests will be added but my intention is to keep what's checked into the main branch always working. I'll make a new branch How, exactly, do you run the tests that you run before you post to pypi so that I can integrate it properly? >> Here's what I'd like to see it do: >> >> [snip...] >> for each product to be tested: >> for each Python version this product is to be tested on: >> install the product >> run its test suite >> record any failures >> > > Sounds good too ! Notice that we also need to try those tests in > various environments, besides python versions: virtualenv, no virtualenv, setuptools present before > we start, not present, etc Yes, I understand what might be possible at some point in the future but, for now, I just want to get the machinery running and in place to support all of those different scenarios at some point in the future. While I'd love to do a comprehensive test suite, I just don't have the time to devote to getting every combination and permutation done it in one shot so my intention is to set up the hooks, get some useful tests into the process that cover recent issues, and make it so that anyone can add new tests with a minimum of configuration. So...what command should I hook it into so I can quickly get this into the testing process before the next release? Thanks, S From LuHe at gmx.at Sun Dec 20 20:44:16 2009 From: LuHe at gmx.at (Lukas Hetzenecker) Date: Sun, 20 Dec 2009 20:44:16 +0100 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <5b8d13220912170119m18f6ac91t717ac1fca79545df@mail.gmail.com> References: <200912131508.36740.LuHe@gmx.at> <200912170851.04694.LuHe@gmx.at> <5b8d13220912170119m18f6ac91t717ac1fca79545df@mail.gmail.com> Message-ID: <200912202044.16433.LuHe@gmx.at> I found a simpler way to do this - Is there any drawback if I try to solve this problem using this way: dist = setup(name='series60-remote', [...] scripts=['series60-remote'] ) # HACK! Copy extra files if dist.have_run.get('install'): install = dist.get_command_obj('install') # Copy textfiles in site-package directory for file in textfiles: install.copy_file(file, os.path.join(install.install_lib, app_dir)) # Copy .sis files on Unix-like systems to /usr/share/series60-remote, on Windows systems # to PREFIX/site-packages/series60_remote/mobile if os.name == 'posix': dest = os.path.join(install.install_data, 'share', install.config_vars['dist_name']) else: dest = os.path.join(install.install_lib, app_dir, 'mobile') install.mkpath(dest) for file in sisfiles: install.copy_file(file, dest) I attached the whole setup.py script too Thank you for your helpful suggestions, Lukas > No, at least not using public API. > > What you can do is to override the install command (the one which > knows the computed install prefix), and use it from there. But you > still cannot control this from the setup.py, that is your command is > called by distutils, and you don't control it. > > David > -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py Type: text/x-python Size: 3280 bytes Desc: not available URL: From david.lyon at preisshare.net Sun Dec 20 23:42:50 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sun, 20 Dec 2009 17:42:50 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: Message-ID: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> Hi Steffen, On Sun, 20 Dec 2009 15:21:04 +0000 (UTC), Steffen Mueller wrote: > It'll be no secret to anyone reading this that there has been a lot of > discussion recently about a CPAN equivalent for Python, sparked by > Guido's "People want CPAN" post to the python-distutils-devel list. Yes Guido says it's a good idea. Users say it's a good idea, but not everybody else shares that view.. it's a hard ask.. > I'm from the Perl crowd[1] and would simply like to share my > late comer's view of how the CPAN came about. Sure it is interesting. > My thesis is that the huge success of the CPAN has been facilitated by > two factors[2]. The first is simplicity. Totally true. That's what python-insiders find it difficult to grasp. > When Jarkko Hietaniemi > originally came up with it, the CPAN was (and mostly still is) just an > FTP archive with a by-author directory structure that is mirrored many > times. Everything else is built on this flexible foundation and has > grown over time. Yes. Pypi is not so dissimilar. The difference is that while CPAN kept its simplicity, python on it's growth phase has taken cheese-shop, added sloppy noodles, and evolved into a bowl of spaghetti. Not only that, it's a bowl of spaghetti with multiple noodle types now, setuptools, distribute, pip etc. > The CPAN specifically does NOT have an official web > service or any kind of development platform. Apart from the directory > structure of the CPAN, the only other key ingredient was the "Perl > Authors Upload SErver" (PAUSE) that handles credentials of the authors, > permissions for namespaces, and serves as a single entry-point for > uploads to the CPAN. PAUSE scans incoming distributions for meta > information and generates in index of modules (namespaces/classes) and > distributions that is itself distributed via the simple CPAN mechanism. Pypi has all that too.. as you say.. > Let me repeat: Everything else is just sugar on top. Sugar for CPAN yes. But in the python world we get just ghee as the sweetener. It just has a different taste. > Specifically, > everything else is sugar provided by *third parties*. Andreas K?nig > wrote and still maintains PAUSE and the CPAN.pm client[3]. Later, Jos > Boumans set out to write the CPANPLUS client. Graham Barr wrote and > still maintains the search.cpan.org website. Randy Kobes wrote the > similar but equally non-official kobesearch.cpan.org. It's nice when you have community spirit like that. > By virtue of the simple design and the (somewhat limited) > published meta-data in form of the simple module index, everyone had > the opportunity to write tools that interface with it. Another (quite) important contrasting difference in python... > There was no > need to "get it all right" from the get-go. Things evolved and we now > have a best-of-breed. The various services by various people are > loosely intertwined[5]. There's a different philosophy in python. I think the management style is more hierarchical. > There is also very little regulation on what is uploaded to CPAN, but > curiously little abuse. I think that is because the majority of people > who are willing to share their work with others free of charge aren't > the type who'd want to crap on other people's front yard. Same with pypi. > My firm belief is that the second most important factor to the success > of the CPAN is people. There are some individuals who have managed > herculean amounts of work and have shown incredible dedication over > years. But it's the combination of a lot of people's work that is more > than its sum. Exactly. Python development philosophy is much more fragmented. In python we just have bike-sheds. It would be good to see leadership pushing people together in a cohesive way, towards common goals. > I'd say the core of the CPAN toolchain gang is not that > large. Depending on where you draw the line, there are maybe 10-50 of > them. That's the ideal size. > Not all, but many of these people have known each other > personally for years from attending the many YAPCs and Perl workshops. > Reading the "People want CPAN" thread, it seemed to me that folks are > fighting each other quite a bit and not always only on technical > grounds. You got it. > On the other hand, it's hard to imagine fighting with somebody > as friendly and welcoming as, for example, Andreas K?nig. Meeting > in person helps this tremendously. Discover the common goals and > agree on the means over a beer. I too have been beer drinking with the perl crowd. Such a great time. Those guys are just so friendly... > So if anyone was to accept a single concrete suggestion from me, it'd > be: "Enable the right heads to get together over a beer." haha. What a notion... > It is the bonds between individuals that can make it all work > well. The success > of the CPAN is due to cultural aspects at least as much as due to its > design and technical merits. While I agree 100%, we're back to the circular argument as to whether we even want something like cpan. Guido says one thing, others another, and users something different. > Please keep in mind that this is only my personal opinion. Thanks for > reading. I hope this can add some perspective. Yes but you're close on 100% right on every point as far as I can see. All you're really saying is that perl developers are happier after having a beer together. And that this doesn't happen as much in python. If that's what you're implying, then I would have to say that its just so true. Python is different.. David From pje at telecommunity.com Mon Dec 21 03:14:14 2009 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 20 Dec 2009 21:14:14 -0500 Subject: [Distutils] Packaging of files, which aren't in the package root folder In-Reply-To: <200912202044.16433.LuHe@gmx.at> References: <200912131508.36740.LuHe@gmx.at> <200912170851.04694.LuHe@gmx.at> <5b8d13220912170119m18f6ac91t717ac1fca79545df@mail.gmail.com> <200912202044.16433.LuHe@gmx.at> Message-ID: <20091221021419.041313A40A2@sparrow.telecommunity.com> At 08:44 PM 12/20/2009 +0100, Lukas Hetzenecker wrote: >I found a simpler way to do this - Is there any drawback if I try to solve >this problem using this way: Yes. The --record option to the install command will be broken (due to not including the directly-copied files). (And that's only the *first* problem that comes to mind. I'm pretty sure there are others.) From regebro at gmail.com Mon Dec 21 11:13:31 2009 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 21 Dec 2009 11:13:31 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> Message-ID: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> What nobody still fails to explain in this discussion is what CPAN "is" and Why Python doesn't already have it. There is just a lot of "CPAN is great!" And "Python needs CPAN" but noone can come up with one single thing that CPAN does that Python doens't have, or explain why CPAN is so great, where PyPI isn't. And unless somebody can do that, this discussion ain't going nowhere. :) A simple place where you can upload packages? Check. An index of those packages? Check. Loads of sugar on top? Check. OK, one drawback there is that Python has sugar on top of the sugar, and slightly competing sugar, and it's difficult to know which type of sucrose to use. But that is being worked on. So... what? If Python needs CPAN, then what is CPAN? Because as far as I can tell, Python has CPAN, and has had it for years. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From ziade.tarek at gmail.com Mon Dec 21 12:14:49 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 21 Dec 2009 12:14:49 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> Message-ID: <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> On Sat, Dec 19, 2009 at 11:05 PM, Tarek Ziad? wrote: > On Sat, Dec 19, 2009 at 10:12 PM, Floris Bruynooghe > wrote: >> On Fri, Dec 18, 2009 at 03:41:48PM +0100, Tarek Ziad? wrote: >>> The remaining point I can see is about : Repository-URL, >>> Repository-Browse-URL, Bug-Tracker-URL. >>> >>> As Ian suggested I propose to change these three fields to: >>> >>> Project-URL (multiple-use) : >>> >>> ? ? A string containing an URL for the project and a label for it. >>> separated by a comma >> [...] I've updated the PEP accordingly. Here's the mail I'll send to python-dev for PEP 345, if anyone sees a problem or something to add: ================= Hi, On behalf of the Distutils-SIG, I would like to propose to addition of PEP 345 (once and *if* PEP 376 is accepted). It's the metadata v1.2: http://www.python.org/dev/peps/pep-0345/ PEP 345 was initiated a while ago by Richard Jones, and reworked by Tres Seaver, I, and some other folks last year at Pycon (Sorry I lost track of the full list of contributors, but if anyone wants to be mentioned in the Acknowledgment section let me know). Then it was reworked throughout the year with PEP 386, in Distutils-SIG, with the help of numerous folks. The major enhancements are: - being able to express dependencies on other *distributions* names, rather than packages names or modules names. This enhancement comes from Setuptools and has been used successfully for years by the community. - being able to express some fields which values are specific to some platforms. For example, being able to define "pywin32" as a dependency *only* on win32. This enhancement will allow any tool to query PyPI and to get the metadata for a particular execution context, without having to download, build, or install the project itself. - being able to provide a list of browsable URLs for the project, like a bug tracker, a repository etc, in addition to the home url. This will allow UIs like PyPI to display a list of URLs for a project. A side-effect will be that a project maintainer will be able to drive its end users to the right places when they need to find detailed documentation or provide some feedback. This enhancement was driven by the discussions about the rating/comment system at PyPI on catalog-sig. We believe that having PEP 376 and PEP 345 accepted will be a major improvement for the Python packaging eco-system. As a side note, I would really like to see them accepted before the first beta of Python 2.7 so we can add these features in 2.7/3.2 and start to work on third-party tools (Distribute, Pip, a standalone version of Distutils for 2.6/2.5, etc..) to get ready to support them by the time 2.7 final is out. Regards Tarek ================ From ziade.tarek at gmail.com Mon Dec 21 12:23:24 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 21 Dec 2009 12:23:24 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> Message-ID: <94bdd2610912210323n54803394k6efa09b23b1c1f34@mail.gmail.com> On Mon, Dec 21, 2009 at 12:14 PM, Tarek Ziad? wrote: [..] > > We believe that having PEP 376 and PEP 345 accepted will be a major > improvement for the Python packaging eco-system. s/PEP 376/PEP 386 From python at davidrobins.net Mon Dec 21 12:42:47 2009 From: python at davidrobins.net (David Robins) Date: Mon, 21 Dec 2009 03:42:47 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> Message-ID: <20091221114247.GB23017@rivendell.internal> On Mon, Dec 21, 2009 at 11:13:31AM +0100, Lennart Regebro wrote: > What nobody still fails to explain in this discussion is what CPAN > "is" and Why Python doesn't already have it. > There is just a lot of "CPAN is great!" And "Python needs CPAN" but > noone can come up with one single thing that CPAN does that Python > doens't have, or explain why CPAN is so great, where PyPI isn't. > > And unless somebody can do that, this discussion ain't going nowhere. :) Here are a couple things I really like about CPAN: 1. Module documentation - the perldoc is extracted, formatted as HTML, and is available for browsing (e.g., search.cpan.org - perhaps this is part of the "sugar" described by Steffen but it tastes delicious). The same could presumably be done with pydoc. (Some modules have some documentation on PyPI, but it's not the pydoc, just a summary.) (The local pydoc server also doesn't help me for modules that I don't have installed yet, and installing every module matching, say, "ID3", and then reading the pydoc is a significant hurdle.) Slightly tangentially, the Python community doesn't seem to have instilled the same documentation culture as the Perl folk. The standard perldoc sections (DESCRIPTION, SYNOPSIS, etc.) are enormously helpful, whereas pydoc seems limited to very dry docstrings, and tends to import unneeded extras (e.g., when 'pydoc dbus' shows the dbus.Array class, it also feels a need to list all the methods of the __builtin__.list class from which it inherits). It's become more of a rule than an exception to have to examine the module source to determine how to use a Python module. 2. A conceptual link between different versions of the same module. On CPAN (search.cpan.org), there's a page for module X with a dropdown for the known versions and their release dates, which may also be downloaded. PyPI appears to treat multiple versions of the same package as completely different entries. A link to an extracted changelog is also convenient. 3. Index by module name (as well as package name). Even further, it would help predictability to make the two match when possible (as perl module X::Y version V will usually be X-Y-V.tar.gz), or at least obviate the need to display package names. Frequently I don't care about the cutesy package name, just what it implements. 4. Namespaces and some way of reserving them. There are likely many modules named postgresql on PyPI, but there's only one DBD::Pg (although there are other PostgreSQL modules that implement the perl DBI driver interface). This also helps with specifying dependencies. > A simple place where you can upload packages? Check. > An index of those packages? Check. > Loads of sugar on top? Check. OK, one drawback there is that Python > has sugar on top of the sugar, and slightly competing sugar, and it's > difficult to know which type of sucrose to use. But that is being > worked on. If Python has the listed items - if they're in the "sugar on top of the sugar" - I apologize, and please point me to them, since I've had a hard time finding them. I don't think competition hurts, either (search.cpan.org vs. kobesearch.cpan.org). > So... what? If Python needs CPAN, then what is CPAN? Because as far as > I can tell, Python has CPAN, and has had it for years. It definitely appears to have the framework, but lacks some finishing touches that would enormously enhance usability. From regebro at gmail.com Mon Dec 21 20:16:27 2009 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 21 Dec 2009 20:16:27 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <20091221114247.GB23017@rivendell.internal> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> Message-ID: <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> On Mon, Dec 21, 2009 at 12:42, David Robins wrote: > 1. Module documentation - the perldoc is extracted, formatted as HTML, > and is available for browsing (e.g., search.cpan.org - perhaps this is > part of the "sugar" described by Steffen but it tastes delicious). The > same could presumably be done with pydoc. (Some modules have some > documentation on PyPI, but it's not the pydoc, just a summary.) > (The local pydoc server also doesn't help me for modules that I don't > have installed yet, and installing every module matching, say, "ID3", > and then reading the pydoc is a significant hurdle.) Interesting. I generally don't find automatically generated documentation useful, but if it works for PERL I guess that's a personal taste. There is a place for module documentation now, but it's not automatically extracted, but a place for any sort of documentation. An example: http://packages.python.org/distribute A site that automatically generated pydoc documentation for all modules on PyPI should be possible. > 2. A conceptual link between different versions of the same module. On > CPAN (search.cpan.org), there's a page for module X with a dropdown for > the known versions and their release dates, which may also be > downloaded. PyPI appears to treat multiple versions of the same package > as completely different entries. A link to an extracted changelog is also > convenient. All "visible" versions are listed on PyPI, it's just that the default setting is to hide old versions by default. I agree that a "show old versions" link could be a good idea there. > 3. Index by module name (as well as package name). Even further, it > would help predictability to make the two match when possible (as perl > module X::Y version V will usually be X-Y-V.tar.gz), or at least obviate > the need to display package names. Frequently I don't care about the > cutesy package name, just what it implements. It's very unusual that these differs in my experience. On the top of my head I can think of two: python-dateutils and Plone, which you use with "import dateutils" and "import Products.CMFPlone" respectively. In the Plone case there is a good reason. :) > 4. Namespaces and some way of reserving them. There are likely many > modules named postgresql on PyPI, but there's only one DBD::Pg (although > there are other PostgreSQL modules that implement the perl DBI driver > interface). This also helps with specifying dependencies. There is no reserving in the Python world, that's true. I'm skeptical. There's enough problems with packages not being updated, we don't want that for namespaces too, that would be terrible. > It definitely appears to have the framework, but lacks some finishing > touches that would enormously enhance usability. So far we have identified a "show all versions" link, and automatically generated documentation. Thanks! Finally something concrete! :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From m.van.rees at zestsoftware.nl Mon Dec 21 23:08:20 2009 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Mon, 21 Dec 2009 22:08:20 +0000 (UTC) Subject: [Distutils] Not picking Older version eggs References: Message-ID: devyan parmar, on 2009-12-18: > > Hi, > > I am using buildout for my project development as well as for deployment. > which makes process easy and help in repeatability. > but facing one problem while choosing particular versions of eggs while > running buildout. > i have eggified my python packages and releasing version wise. > i have created "versions.cfg" where i mentioned all my project related > packages with proper version. > it's working fine for latest eggs version. but if i wants to choose older > version egg from my package server, It takes latest only. > even after specifying in versions.cfg also. > > following attributes am using in my buildout.cfg > > [buildout] > newest = false > prefer-final = true > allow-picked-versions = true > extends = version.cfg > preferences.cfg > versions = versions > > to avoid this i am deleting existing packages from my eggs folder and > rerunning buildout. > > Please suggest me here... I do not see this behaviour. I tried this buildout.cfg: ============================== [buildout] newest = false prefer-final = true allow-picked-versions = true versions = versions parts = virtualenv [versions] zc.buildout = 1.4.3 virtualenv = 1.4.2 zc.recipe.egg = 1.2.2 [virtualenv] recipe = zc.recipe.egg:scripts ============================== If I lower any of the versions, that change gets picked up by the next bin/buildout run. You may need to run buildout once in newest mode on the command line: 'bin/buildout -n'. At least, I sometimes seem to need that when buildout has previously already gotten some version of a package and apparently sometimes does not realize that the required version has changed. Newest mode here effectively says: yes, buildout is allowed to get other versions. -- Maurits van Rees | http://maurits.vanrees.org/ Work | http://zestsoftware.nl/ "This is your day, don't let them take it away." [Barlow Girl] From cournape at gmail.com Mon Dec 21 23:48:55 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 07:48:55 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> Message-ID: <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> On Mon, Dec 21, 2009 at 7:13 PM, Lennart Regebro wrote: > What nobody still fails to explain in this discussion is what CPAN > "is" and Why Python doesn't already have it. That's not the right question to ask. The problem is not much a feature problem as much as a fundamental implementation and "state of mind". Reliable packaging requires explicit handling, where the whole python stack for packaging relies a lot on implicit behavior. I don't know much about CPAN, but both CRAN ("CPAN for R") and hackage ("CPAN for haskell") work reliably where Pypi often fails, and the reasons are easy to see: - distutils (and most tools on top) throw away metadata, and then other tools try to guess them back. That alone is source of numerous errors, weird hacks, and unexplainable issues. Generally, when there is a design error at some level, instead of fixing it at this level, tools try to add another level of complexity to circumvent it. The fact that python has something like 5 or 6 tools to deploy things where most other languages have only one or two is quite striking. - Linked to the above, metadata are not enforced in pypi. This just cannot work. Most other systems in existence enforce strict rules. - Most crucial stages (install by distutils/setuptools, retrieving metadata, file formats like eggs) are not specified, and implemented/executed in an ad-hoc fashion. As a result, easy_install often fails to install things correctly, and the errors are hard to recover for people not familiar with python. For example, when you install a package, the previous installation is not removed. Worse, there is no way to correctly uninstall things, because the way distutils install things is dumb (dumping whatever is in the build directories instead of building a precise list of what is to be installed). This alone is a constant source of issues for us with numpy and scipy. - etc... David From ziade.tarek at gmail.com Tue Dec 22 01:04:28 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 22 Dec 2009 01:04:28 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> Message-ID: <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> On Mon, Dec 21, 2009 at 11:48 PM, David Cournapeau wrote: > On Mon, Dec 21, 2009 at 7:13 PM, Lennart Regebro wrote: >> What nobody still fails to explain in this discussion is what CPAN >> "is" and Why Python doesn't already have it. > > That's not the right question to ask. The problem is not much a > feature problem as much as a fundamental implementation and "state of > mind". Reliable packaging requires explicit handling, where the whole > python stack for packaging relies a lot on implicit behavior. I don't > know much about CPAN, but both CRAN ("CPAN for R") and hackage ("CPAN > for haskell") work reliably where Pypi often fails, and the reasons > are easy to see: > > ?- distutils (and most tools on top) throw away metadata, and then > other tools try to guess them back. That alone is source of numerous > errors, weird hacks, and unexplainable issues. Generally, when there > is a design error at some level, instead of fixing it at this level, > tools try to add another level of complexity to circumvent it. The > fact that python has something like 5 or 6 tools to deploy things > where most other languages have only one or two is quite striking. I think you are quite confused here. The "metadata", as defined in PEP 314, are not thrown away by Distutils, but properly uploaded at PyPI. Other tools like Setuptools created their own metadata extension, that were added in extra files when distributions are created and uploaded. This is not a *design error*, this is called *innovation*. And PEP 345 (the new metadata v1.2) is here to catch up with these changes that are widely used by the community, and will eventually make it simpler for all tools to get these new metadata. http://www.python.org/dev/peps/pep-0345/ > ?- Linked to the above, metadata are not enforced in pypi. This just > cannot work. Most other systems in existence enforce strict rules. I am not sure what you mean by enforcing strict rules. You need to be more precise. What do you want to do ? reject any package that doesn't meet some QA tests ? Is PyPI is a community repository, or an entreprise repository ? > ?- Most crucial stages (install by distutils/setuptools, retrieving > metadata, file formats like eggs) are not specified, and > implemented/executed in an ad-hoc fashion. As a result, easy_install > often fails to install things correctly, and the errors are hard to > recover for people not familiar with python. For example, when you > install a package, the previous installation is not removed. Worse, > there is no way to correctly uninstall things, because the way > distutils install things is dumb (dumping whatever is in the build > directories instead of building a precise list of what is to be > installed). This alone is a constant source of issues for us with > numpy and scipy. I would have understood this frustration a year ago, but now, that's just a free rant/flame :) You can't ignore the work we are doing to address these issues http://www.python.org/dev/peps/pep-0376/ Now, what would really help, is having you (or anyone else) help around in these PEPs, rather than ranting about packaging like that from time to time. Regards Tarek From glyph at twistedmatrix.com Tue Dec 22 01:04:50 2009 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 21 Dec 2009 19:04:50 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> Message-ID: <9ABA6194-D040-4F50-AAEE-ED1B800D430B@twistedmatrix.com> On Dec 21, 2009, at 5:48 PM, David Cournapeau wrote: > On Mon, Dec 21, 2009 at 7:13 PM, Lennart Regebro wrote: >> What nobody still fails to explain in this discussion is what CPAN >> "is" and Why Python doesn't already have it. > > That's not the right question to ask. The problem is not much a > feature problem as much as a fundamental implementation and "state of > mind". Reliable packaging requires explicit handling, where the whole > python stack for packaging relies a lot on implicit behavior. It is definitely the right question to ask, and it is very much a feature problem. The missing feature is "install what I mean". easy_install is missing by default in most cases, and then broken by default when you install it. If a fresh python install, you cannot just type "easy_install foo", or even 'cd foo; python setup.py install' and reliably get a copy of 'foo' installed for whatever user asked for it. Instead you get piles of cryptic error messages until you learn how to use it and what command-line options to pass. Now, CPAN is not perfect, and you can most definitely get cryptic error messages out of it. But, *most of the time*, it just works, and *most of the time*, developers can install dependencies and users can install software without really thinking about it too hard. Everything you're saying about mindset and standardization may be good, and in fact entirely necessary to achieving this goal. But it is very important that, as a community, we: A) keep our eyes on the prize, and try to improve the default, out-of-the-box Python package installation experience wherever possible, and B) be clear about what the prize _is_. It's really important to nail down what it is that we all agree needs to improve. I say this because if someone wants to ask a question like "what is this thing that everyone seems to say we should work on", I think it's important to answer it. In one sense of "not a feature problem", I think you're right. The problem here is not a particular *advanced* feature, some more sophisticated option, although many features might help fix it: the problem is that the user experience of existing functionality is bad. We're not hearing a lot of lucid articulation of what exactly the "CPAN problem" is, and I believe the reason is that when you actually look at the problems and describe them, they're easy to work around. setup.py install doesn't work for users who aren't root? Well, maybe that's not really a feature problem, it's a documentation problem. For most of us it's pretty easy to set your PATH and PYTHONPATH and type --prefix and --single-version-externally-managed, and then everything works fine. Or use something like virtualenv. Or you can just use Python 2.6 and set only your PATH, as long as you know that Windows keeps things in one directory layout (%APPDATA%/Python) and POSIX another (~/.local/lib/site-packages). From david.lyon at preisshare.net Tue Dec 22 01:58:02 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 21 Dec 2009 19:58:02 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> Message-ID: <08cb9b299725da7f39a4b62c50486b06@preisshare.net> On Tue, 22 Dec 2009 07:48:55 +0900, David Cournapeau wrote: > (cpan) ... is not much a > feature problem as much as a fundamental implementation and "state of > mind". Yes. Their state of mind is make it simple and make it work across the board for everything for everyone. > Reliable packaging requires explicit handling, where the whole > python stack for packaging relies a lot on implicit behavior. Yep.. That's it in a nutshell. > (distutils) .. is a design error ... , instead of fixing > it at this level, tools try to add another level of complexity > to circumvent it. This now begs the beer-drinking style (cpan/perl developers understand) question of what to actually do about this in 2010. One thing I'm pretty certain of is that people would prefer to accomplish something and after that drink beer together. That was really the secret of the cpan success. They loved the sense of accomplishment. They loved bragging too - but it was always a welcoming type of bragging that one could easily fall in to and feel a part of. After reading your email David, you leave the reader with no other conclusion other than a new explicit build tool is needed for 2010. Otherwise, we're all going to be sitting helplessly reading the same old 'how can I tell distutils I want to ...?" for the whole of the next decade. At the same time, there won't be much of an effort to address the design issues you talk about. Simply because of legacy software syndrome management issues. What can be done, if there is good delegation of the tasks, is for a new pypi toolset (let's not fragment things and confuse new users with lots of different names) to be started. At the highest level, the toolset needs to: - build python packages/apps from explicit metadata - upload and register them - download and install (and the other functions) I'd toss in a few extra things not in cpan. Like incorporating buildout support. I don't use it myself but so many others do. The bigger issue is about deployment. It's about getting python apps/libraries out to all the machines out there with as little effort as possible. The Package concept is based on a static snapshot idea. But how much python software is now in mercurial or bzr repositories ? lots of times software is slow moving. Yes, we do want that latest bug fix. Say it didn't work.. jump back a revision or two. Is a 'package' really a good way to deploy? I'm not against them, just suggesting that a repository link in pypi metadata is an equally good but perphaps more modern way. What is python doing or going to do to help us manage all these deployment issues. At some point in time the stdlib has got to address lots of 'stuff' landing up on the end machine. Batteries included has just got to include built in bzr or mercurial or cvs or svn support. Just where I work I've written enough mini apps for different machines deployment is getting to be a challenge. Most have scm. But setup from scratch is still un-optimal imo. How does our fundamental distutils design cope with the evolving landscape? including supercomputer and server farm type requirements. Also just plain commercial and scientific (workstation) type implementations. My list of future packaging/configuration/ deployment projects to interoperate with are (but not limited to): - buildout - celery (http://pypi.python.org/pypi/celery/0.3.0) - pip/distribute/setuptools - bzr/svn/hg etc Add to that the web frameworks.. Anyway, you've totally convinced me it's time for a new pypi package/app build tool.. I would like to also suggest that it be built upon some small python web framework like cherrypy and fire up a browser for the user to work with. I don't want another text based command line program sorry. Maybe it is just a django app. Anybody adventurous enough to build a package or an app isn't going to mind installing a small web app to run on port 8080 or whatever to do their package build - imo. Cheers.. drink and be merry David From cournape at gmail.com Tue Dec 22 02:07:07 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 10:07:07 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> Message-ID: <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> On Tue, Dec 22, 2009 at 9:04 AM, Tarek Ziad? wrote: > On Mon, Dec 21, 2009 at 11:48 PM, David Cournapeau wrote: >> On Mon, Dec 21, 2009 at 7:13 PM, Lennart Regebro wrote: >>> What nobody still fails to explain in this discussion is what CPAN >>> "is" and Why Python doesn't already have it. >> >> That's not the right question to ask. The problem is not much a >> feature problem as much as a fundamental implementation and "state of >> mind". Reliable packaging requires explicit handling, where the whole >> python stack for packaging relies a lot on implicit behavior. I don't >> know much about CPAN, but both CRAN ("CPAN for R") and hackage ("CPAN >> for haskell") work reliably where Pypi often fails, and the reasons >> are easy to see: >> >> ?- distutils (and most tools on top) throw away metadata, and then >> other tools try to guess them back. That alone is source of numerous >> errors, weird hacks, and unexplainable issues. Generally, when there >> is a design error at some level, instead of fixing it at this level, >> tools try to add another level of complexity to circumvent it. The >> fact that python has something like 5 or 6 tools to deploy things >> where most other languages have only one or two is quite striking. > > I think you are quite confused here. I really don't think I am. > The "metadata", as defined in PEP 314, > are not thrown away by Distutils, but properly uploaded at PyPI. But there is much more than what PEP 314 defines. Think about everything which is in an egg, for example. How are files locations are described, how are files outside site-packages described, etc... As a concrete example, there is not enough metadata in every installer so that you can convert from one to the other, even though this would be very useful and definitely possible. So it is not just metadata at the "input" stage (although there is certainly a lot of things lost there too), but how they are represented internally, and how they are represented in the various installers. This matters a lot for people who extend distutils. > >> ?- Linked to the above, metadata are not enforced in pypi. This just >> cannot work. Most other systems in existence enforce strict rules. > > I am not sure what you mean by enforcing strict rules. You need to be > more precise. For example, you can upload a distribution which does not even have a name, or a version, easy_install tries to find tarballs by scrapping webpages, etc... > > What do you want to do ? reject any package that doesn't meet some QA tests ? > Is PyPI is a community repository, or an entreprise repository ? That's common sense to reject packages which do not define a basic set of metadata, and this has nothing to do with enterprise vs community. Debian, for example, enforces metadata in its package, and it does not qualify as enterprise software to me if the term is meant as the opposite of community. Those metada are needed, because other tools do use them (e.g. easy_install, and I guess pip as well). > I would have understood this frustration a year ago, but now, that's > just a free rant/flame ?:) I have tried to explain several times about describing how files are installed where, and everytime I was shut down (not by you). In particular, I think I have explained already why I believe PEP 376 does not solve the issue of interoperation with other installers because it loses source/target directory information. David From martin at v.loewis.de Tue Dec 22 02:13:56 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Dec 2009 02:13:56 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> Message-ID: <4B301D54.6050109@v.loewis.de> > FYI. This Distutils-SIG thread is about proposing PEP 345 and impacts PyPI. > So if there's anything that look suspicious to anyone, please join the > discussion at Distutils-SIG In a number of places (Requires-Dist, Requires-Python), comma-separated lists of version constraints are used. The PEP needs to specify what the collective constraint means that gets specified. Most likely, the intended meaning is that the constraints get and-combined, in which case the example Requires-Python: 2.5, 2.6 is non-sensical (no Python release meets that constraint). (else, if it was meant to denote or-combination, then ">1.0, !=1.3.4, <2.0" would be non-sensical, specifying no constrating at all) For the Copyright field, I'm not sure what the purpose of it is. If it is what I think it is (listing attribution), it should probably be specified as "multiple use". For Documentation, I think the entire field must be reconsidered. If it is really meant to be reStructuredText, then the spec should explain how to do leading spaces/indentation. For Platform, I fail to see the point of supporting both multiple use, and comma-separated lists. For Metadata-Version, I think formally, the only legal value according to the PEP is 1.2. If 1.0 and 1.1 are also conforming values, the PEP should elaborate what it means to put a different version number into the field. Regards, Martin From cournape at gmail.com Tue Dec 22 02:24:07 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 10:24:07 +0900 Subject: [Distutils] Specifying eggs and build manifest ? Message-ID: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.com> Hi, I wonder if there is any interest in the current distribute effort to specify some low-level commonalities outside the distutils code for interoperation. In particular, I had two things in mind: 1 Formally specifying the egg format (and versioning it !) - or is egg format outside distribute goal ? 2 Formally specifying something like a build manifest which would be used for installation. I think the first point is self-explanatory. Concerning the second point, the current way of installing things in distutils is a bit too ad-hoc to my taste, and could be fixed inside distutils to make it easier to interoperate with other methods (be it native OS methods or other python-related tools). Right now, distutils install things from a list of files, but this only contains the target location for some kind of files. What about the following kind of format instead (or in complement for backward-compatibility): - split the flat list into a list of "typed" sections, where type would be e.g. data, python files, extensions, scripts, etc... - for each section, specify both the source and target directory. - the format should be easy to parse/produce, and versioned. The rationale is that such a format would enable the following: - it becomes easier to post-process files at install time, with different methods for each type - such a format could form a basis to produce "conventional" install, eggs and other kind of installers, including convertion between them. Assuming this idea fits into what people involved in distribute are doing, I would be willing to propose a more formal description - I already have the implementation in my own project toydist, where I use build manifest to do explicit install and eggs/wininst conversion. cheers, David From david.lyon at preisshare.net Tue Dec 22 02:33:28 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 21 Dec 2009 20:33:28 -0500 Subject: [Distutils] =?utf-8?q?Specifying_eggs_and_build_manifest_=3F?= In-Reply-To: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.com> References: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.com> Message-ID: <64f67793cfd9fefd0a5c59e95eafc0ea@preisshare.net> Just use buildout... On Tue, 22 Dec 2009 10:24:07 +0900, David Cournapeau wrote: > Hi, > > I wonder if there is any interest in the current distribute effort to > specify some low-level commonalities outside the distutils code for > interoperation. In particular, I had two things in mind: > 1 Formally specifying the egg format (and versioning it !) - or is > egg format outside distribute goal ? > 2 Formally specifying something like a build manifest which would be > used for installation. > > I think the first point is self-explanatory. Concerning the second > point, the current way of installing things in distutils is a bit too > ad-hoc to my taste, and could be fixed inside distutils to make it > easier to interoperate with other methods (be it native OS methods or > other python-related tools). Right now, distutils install things from > a list of files, but this only contains the target location for some > kind of files. What about the following kind of format instead (or in > complement for backward-compatibility): > > - split the flat list into a list of "typed" sections, where type > would be e.g. data, python files, extensions, scripts, etc... > - for each section, specify both the source and target directory. > - the format should be easy to parse/produce, and versioned. > > The rationale is that such a format would enable the following: > - it becomes easier to post-process files at install time, with > different methods for each type > - such a format could form a basis to produce "conventional" install, > eggs and other kind of installers, including convertion between them. > > Assuming this idea fits into what people involved in distribute are > doing, I would be willing to propose a more formal description - I > already have the implementation in my own project toydist, where I use > build manifest to do explicit install and eggs/wininst conversion. > > cheers, > > David > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From jari.pennanen at gmail.com Mon Dec 21 23:16:44 2009 From: jari.pennanen at gmail.com (Jari Pennanen) Date: Tue, 22 Dec 2009 00:16:44 +0200 Subject: [Distutils] Swig build_py ran before build_ext Message-ID: <6d0569fc0912211416p63c8e106na3fafe80cbf22589@mail.gmail.com> Hi! I have old/new problem, once mentioned in 2004 at this same mailing list ( http://mail.python.org/pipermail/distutils-sig/2004-March/003807.html ) but surely that implementation is bad... But the problem persists, I use SWIG and had to Google all over to only find out build order cannot be changed. Someone should implement build_order keyword argument to setup, e.g.: setup(..., build_order=['build_ext', 'build_py']) If some orders are missing, like in above example: build_clib, build_scripts, they would be appended as last. Anyways, the point is make the build order customizable through setup keyword. From ziade.tarek at gmail.com Tue Dec 22 02:50:38 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 22 Dec 2009 02:50:38 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> Message-ID: <94bdd2610912211750q416e2972o1b28fb4103045214@mail.gmail.com> On Tue, Dec 22, 2009 at 2:07 AM, David Cournapeau wrote: [..] > >> The "metadata", as defined in PEP 314, >> are not thrown away by Distutils, but properly uploaded at PyPI. > > But there is much more than what PEP 314 defines. Then that's not the metadata as defined in python packaging, but some other information you are talking about. >Think about > everything which is in an egg, for example. How are files locations > are described, how are files outside site-packages described, etc... > As a concrete example, there is not enough metadata in every installer > so that you can convert from one to the other, even though this would > be very useful and definitely possible. > So it is not just metadata at the "input" stage (although there is > certainly a lot of things lost there too), but how they are > represented internally, and how they are represented in the various > installers. This matters a lot for people who extend distutils. I think you are talking about internal representation of elements in an archive built by distutils here, which is different from the metadata that describes the project/release and pushed at PyPI. There's no need to publish and share those extra information unless you want to create some kind of converter. What's important is to clearly define the metadata of a project, as published at PyPI and installed on your system. Not the internal cooking of each installer. PEP 376 as a matter of fact, provides API to read/write the metadata of installed distributions. Now, for extending distutils, that's another problem: we already said that distutils was hard to extend, and we have proposed some changes for this (for example to have a configure/build/install scheme), but this has nothing to do with the published metadata. As a matter of fact, I am working on a configure command for distutils, looking at 4suite's work I've started a prototype here : http://bitbucket.org/tarek/distutils-configure (early stage) you are welcome to join :) > >> >>> ?- Linked to the above, metadata are not enforced in pypi. This just >>> cannot work. Most other systems in existence enforce strict rules. >> >> I am not sure what you mean by enforcing strict rules. You need to be >> more precise. > > For example, you can upload a distribution which does not even have a > name, or a version, easy_install tries to find tarballs by scrapping > webpages, etc... Two things: 1/ you get warnings if you try to build a distribution with missing metadata and there's now a command in 2.7 called "check" so you can set a strict control on client side 2/ easy_install doesn't "scrap" webpages but uses a protocol implemented by PyPI - see http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api > >> >> What do you want to do ? reject any package that doesn't meet some QA tests ? >> Is PyPI is a community repository, or an entreprise repository ? > > That's common sense to reject packages which do not define a basic set > of metadata, and this has nothing to do with enterprise vs community. > Debian, for example, enforces metadata in its package, and it does not > qualify as enterprise software to me if the term is meant as the > opposite of community. I am even opposing PyPI to debian packages because PyPI allows anyone to upload some python code to share, even if it doesn't have all the required metadata or QA. Installers are not the only PyPI end users right now. IOW, you don't have to be a professional packager to publish and share code at PyPI. That's how I understand its philosophy and I think it goes pretty well with Python's own philosophy. > >> I would have understood this frustration a year ago, but now, that's >> just a free rant/flame ?:) > > I have tried to explain several times about describing how files are > installed where, and everytime I was shut down (not by you). In > particular, I think I have explained already why I believe PEP 376 > does not solve the issue of interoperation with other installers > because it loses source/target directory information. Are you referring to the install schemes ? these schemes are described in the stdlib itself, but are not as detailed as what you could find in Debian for sure. And these scheme only concerns Python locations, because other locations were a bit out of distutils scope. But some people are working on extending that in PEP 376 so we can point stuff like man pages etc. But distutils will never ever compete with debian's dpkg or fedora's rpm if you need a full blown packaging system for a given platform. Rather, the best thing we can do is to make sure those packagers can easily extract info out of distutils distributions; if you think this can be improved, if you have examples of info loss, as you said that's the perfect thing to throw in the PEP discussions. And if you already did some months ago, I am sorry, I think you will have to do it again because that just means imho that the "distutils community" was not ready to hear it back then. But it grew up a little bit since then so I think it worth a try. Tarek -- Tarek Ziad? | http://ziade.org From cournape at gmail.com Tue Dec 22 02:58:48 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 10:58:48 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912211750q416e2972o1b28fb4103045214@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <94bdd2610912211750q416e2972o1b28fb4103045214@mail.gmail.com> Message-ID: <5b8d13220912211758r7b83c0e3q5ecea613c590eb65@mail.gmail.com> On Tue, Dec 22, 2009 at 10:50 AM, Tarek Ziad? wrote: > Two things: > > 1/ you get warnings if you try to build a distribution with missing metadata > ? ?and there's now a command in 2.7 called "check" so you can set a > strict control > ? ?on client side This is nice, I missed this new check feature. This is definitely a step in the right direction. > Are you referring to the install schemes ? these schemes are described > in the stdlib itself, but are not as detailed as what you could find > in Debian for sure. I have started a new thread with a more details explanation, since this is not directly related to PyPi (although it is important to make installing from Pypi more robust IMHO). cheers, David From cournape at gmail.com Tue Dec 22 03:04:05 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 11:04:05 +0900 Subject: [Distutils] Swig build_py ran before build_ext In-Reply-To: <6d0569fc0912211416p63c8e106na3fafe80cbf22589@mail.gmail.com> References: <6d0569fc0912211416p63c8e106na3fafe80cbf22589@mail.gmail.com> Message-ID: <5b8d13220912211804s1752092bk2518ee616dfdc211@mail.gmail.com> On Tue, Dec 22, 2009 at 7:16 AM, Jari Pennanen wrote: > Hi! > > I have old/new problem, once mentioned in 2004 at this same mailing > list ( http://mail.python.org/pipermail/distutils-sig/2004-March/003807.html > ) but surely that implementation is bad... But the problem persists, I > use SWIG and had to Google all over to only find out build order > cannot be changed. > > Someone should implement build_order keyword argument to setup, e.g.: > > ? ?setup(..., build_order=['build_ext', 'build_py']) > > If some orders are missing, like in above example: build_clib, > build_scripts, they would be appended as last. Anyways, the point is > make the build order customizable through setup keyword. This would need to be tested/confirmed on real packages, but I am afraid it would break a lot of packages, especially the ones extending distutils. cheers, David From ziade.tarek at gmail.com Tue Dec 22 03:06:58 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 22 Dec 2009 03:06:58 +0100 Subject: [Distutils] Specifying eggs and build manifest ? In-Reply-To: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.com> References: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.com> Message-ID: <94bdd2610912211806t43976e63h270414aa963f5282@mail.gmail.com> On Tue, Dec 22, 2009 at 2:24 AM, David Cournapeau wrote: > Hi, > > I wonder if there is any interest in the current distribute effort to > specify some low-level commonalities outside the distutils code for > interoperation. In particular, I had two things in mind: > ?1 Formally specifying the egg format (and versioning it !) - or is > egg format outside distribute goal ? If you are referring to the installation format, Distribute wants to drop any extra format in the future and stick with a unique, standard format described in PEP 376 to avoid having an heterogeneous site-packages. > ?2 Formally specifying something like a build manifest which would be > used for installation. > > I think the first point is self-explanatory. Concerning the second > point, the current way of installing things in distutils is a bit too > ad-hoc to my taste, and could be fixed inside distutils to make it > easier to interoperate with other methods (be it native OS methods or > other python-related tools). Right now, distutils install things from > a list of files, but this only contains the target location for some > kind of files. What about the following kind of format instead (or in > complement for backward-compatibility): > > ?- split the flat list into a list of "typed" sections, where type > would be e.g. data, python files, extensions, scripts, etc... > ?- for each section, specify both the source and target directory. > ?- the format should be easy to parse/produce, and versioned. I like the idea > > The rationale is that such a format would enable the following: > ?- it becomes easier to post-process files at install time, with > different methods for each type > ?- such a format could form a basis to produce "conventional" install, > eggs and other kind of installers, including convertion between them. > > Assuming this idea fits into what people involved in distribute are > doing, I would be willing to propose a more formal description - I > already have the implementation in my own project toydist, where I use > build manifest to do explicit install and eggs/wininst conversion. Distribute 0.7 is still under heavy construction, and the goals we have about the installation features are not fully defined yet. What we want for sure is to make it the laboratory of Distutils to improve things in smaller cycles than python. And having a better build description sounds like something we could use/have. From ziade.tarek at gmail.com Tue Dec 22 03:16:26 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 22 Dec 2009 03:16:26 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912211758r7b83c0e3q5ecea613c590eb65@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <94bdd2610912211750q416e2972o1b28fb4103045214@mail.gmail.com> <5b8d13220912211758r7b83c0e3q5ecea613c590eb65@mail.gmail.com> Message-ID: <94bdd2610912211816i6f034e53j3b75163340404f65@mail.gmail.com> On Tue, Dec 22, 2009 at 2:58 AM, David Cournapeau wrote: > On Tue, Dec 22, 2009 at 10:50 AM, Tarek Ziad? wrote: > >> Two things: >> >> 1/ you get warnings if you try to build a distribution with missing metadata >> ? ?and there's now a command in 2.7 called "check" so you can set a >> strict control >> ? ?on client side > > This is nice, I missed this new check feature. This is definitely a > step in the right direction. > more on this: http://docs.python.org/dev/distutils/examples.html#checking-a-package From pje at telecommunity.com Tue Dec 22 03:42:49 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 21 Dec 2009 21:42:49 -0500 Subject: [Distutils] Specifying eggs and build manifest ? In-Reply-To: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.co m> References: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.com> Message-ID: <20091222024255.B73E73A4175@sparrow.telecommunity.com> At 10:24 AM 12/22/2009 +0900, David Cournapeau wrote: > 1 Formally specifying the egg format (and versioning it !) - or is >egg format outside distribute goal ? FYI: http://peak.telecommunity.com/DevCenter/EggFormats From cournape at gmail.com Tue Dec 22 03:48:39 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 11:48:39 +0900 Subject: [Distutils] Specifying eggs and build manifest ? In-Reply-To: <20091222024255.B73E73A4175@sparrow.telecommunity.com> References: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.com> <20091222024255.B73E73A4175@sparrow.telecommunity.com> Message-ID: <5b8d13220912211848q2221a61ds3a59c5075020825d@mail.gmail.com> On Tue, Dec 22, 2009 at 11:42 AM, P.J. Eby wrote: > At 10:24 AM 12/22/2009 +0900, David Cournapeau wrote: >> >> ?1 Formally specifying the egg format (and versioning it !) - or is >> egg format outside distribute goal ? > > FYI: http://peak.telecommunity.com/DevCenter/EggFormats I know about this page, but it says "Instead, this is internal documentation for how those concepts and features are implemented in concrete terms. It is intended for people who are working on the setuptools code base, who want to be able to troubleshoot setuptools problems, want to write code that reads the file formats involved, or want to otherwise tinker with setuptools-generated files and directories." So unless this description is not accurate anymore, it is not a specification. I want to build eggs, without having distutils (and setuptools) involved at all. I would like to be able to do so while staying compatible with the existing infrastructure, so that if someone use my own tool instead of setuptools to produce an egg, it still works as expected with virtualenv, buildout, etc... David From cournape at gmail.com Tue Dec 22 04:21:07 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 12:21:07 +0900 Subject: [Distutils] Specifying eggs and build manifest ? In-Reply-To: <94bdd2610912211806t43976e63h270414aa963f5282@mail.gmail.com> References: <5b8d13220912211724i4291e0d9la3a4433166a5c435@mail.gmail.com> <94bdd2610912211806t43976e63h270414aa963f5282@mail.gmail.com> Message-ID: <5b8d13220912211921j44ad4822g35ddb3557c215eb7@mail.gmail.com> On Tue, Dec 22, 2009 at 11:06 AM, Tarek Ziad? wrote: > If you are referring to the installation format, Distribute wants to > drop any extra format in the future and stick with a unique, standard > format described in PEP 376 to avoid having an heterogeneous > site-packages. Does that mean that eggs would be "deprecated" ? > I like the idea Cool :) Here is what I got so far. I have my small own packaging solution, and a small utility called toymaker, which works as steps (toymaker configure, toymaker build, etc...). For this discussion, the only thing to know is that toymaker build produces this build manifest, and both toymaker install/build_egg read this build manifest to produce the install target. Right now, the format I use is ad-hoc, what matters at this point is the information it describes, and anything is open to changes if anyone has a better idea on a particular point. There are three sections: - A copy of PKG-INFO data in a simple form for easy parsing - A variable section, containing paths (bindir, libdir, etc..) and executables - A files section, which contains subsections of files to install. For example, for sphinx, toymaker build produces this name=Sphinx license=BSD author=Georg Brandl url=http://sphinx.pocoo.org/ top_levels=sphinx download_url=http://pypi.python.org/pypi/Sphinx summary=Python documentation generator author_email=georg at python.org version=0.6.3 platforms=any install_requires=Pygments>=0.8 install_requires=Jinja2>=2.1 install_requires=docutils>=0.4 classifiers=Development Status :: 4 - Beta classifiers=Environment :: Console classifiers=Environment :: Web Environment classifiers=Intended Audience :: Developers classifiers=License :: OSI Approved :: BSD License classifiers=Operating System :: OS Independent classifiers=Programming Language :: Python classifiers=Topic :: Documentation classifiers=Topic :: Utilities description=Sphinx is a tool that makes it easy to create intelligent and beautiful description=documentation for Python projects (or other documents consisting of description=multiple reStructuredText sources), written by Georg Brandl. description=It was originally created to translate the new Python documentation, description=but has now been cleaned up in the hope that it will be useful to many description=other projects. description= description=Sphinx uses reStructuredText as its markup language, and many of its strengths description=come from the power and straightforwardness of reStructuredText and its description=parsing and translating suite, the Docutils. description= description=Although it is still under constant development, the following features description=are already present, work fine and can be seen "in action" in the Python docs: description= description=* Output formats: HTML (including Windows HTML Help), plain text and LaTeX, description= for printable PDF versions description=* Extensive cross-references: semantic markup and automatic links description= for functions, classes, glossary terms and similar pieces of information description=* Hierarchical structure: easy definition of a document tree, with automatic description= links to siblings, parents and children description=* Automatic indices: general index as well as a module index description=* Code handling: automatic highlighting using the Pygments highlighter description=* Various extensions are available, e.g. for automatic testing of snippets description= and inclusion of appropriately formatted docstrings. description= description=A development egg can be found `here description=`_. !- VARIABLES paths pkgname=Sphinx datarootdir=$prefix/share eprefix=$prefix sitedir=$libdir/python$py_version_short/site-packages libdir=$eprefix/lib prefix=/usr/local py_version_short=2.6 gendatadir=$sitedir datadir=$datarootdir includedir=$prefix/include sbindir=$eprefix/sbin pkgdatadir=$datadir/$pkgname bindir=$eprefix/bin mandir=$datarootdir/man executables sphinx-build=sphinx:main sphinx-quickstart=sphinx.quickstart:main sphinx-autogen=sphinx.ext.autosummary.generate:main !- FILELIST executables:sphinx-build srcdir=build/scripts-2.6 target=$bindir sphinx-build executables:sphinx-quickstart srcdir=build/scripts-2.6 target=$bindir sphinx-quickstart executables:sphinx-autogen srcdir=build/scripts-2.6 target=$bindir sphinx-autogen datafiles:gendata_sphinx_locale ... The file contains just enough data to build egg and install on disk ATM, and I include the build manifest in the produced egg so that conversion to wininst-based (or other types of) installers is possible. For the file list, each section has the same structure: - type:name -> type would be used for post-install (byte compile python files, chmod binaries, etc...), name is not really important, but it enables having several subsections of the same type (so that different target/srcdir are possible) - the next two lines are srcdir and target: target can be a variable as defined in the path section (this is important to "relocate" installs in some cases), srcdir is such as every file of the section can be found in srcdir/file. Although I have not thought about the problem yet, it should also be usable for uninstall, and for on-disk query of installed packages. I should note at this point that it is mostly a rip-off of one concept in cabal I really like, the installed package information, which is used for those purposes as well: http://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/Distribution-InstalledPackageInfo.html > Distribute 0.7 is still under heavy construction, and the goals we have > about the installation features are not fully defined yet. What we want for > sure is to make it the laboratory of Distutils to improve things in smaller > cycles than python. And having a better build description sounds like > something we could use/have. I could start implementing it within distutils/Distribute, at least for the parts which I guess will be the least controversial (the FILE section and the path variables), and specify the exact file format. I think it should be possible for the install command of distribute to only rely on this file - this would actually be a good test, comparing installs with the old and the new way. David From regebro at gmail.com Tue Dec 22 08:43:05 2009 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 22 Dec 2009 08:43:05 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> Message-ID: <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> On Mon, Dec 21, 2009 at 23:48, David Cournapeau wrote: > mind". Reliable packaging requires explicit handling, where the whole > python stack for packaging relies a lot on implicit behavior. Can you give examples of this implicit behaviour? > ?- distutils (and most tools on top) throw away metadata, and then > other tools try to guess them back. What metadata is thrown away? > tools try to add another level of complexity to circumvent it. The > fact that python has something like 5 or 6 tools to deploy things > where most other languages have only one or two is quite striking. Can you list these 5 or 6 tools to deploy, because I'm only aware of two, easy_install and pip. And in the current refactoring of Distribute, easy_install is going away. > ?- Linked to the above, metadata are not enforced in pypi. This just > cannot work. Most other systems in existence enforce strict rules. It is working, right now, so that's a strange claim. Obviously, you are less likely to use a package with less metadata, but that's the package authors problem, not your, or? > For example, when you > install a package, the previous installation is not removed. Worse, > there is no way to correctly uninstall things It's not worse, it's the same. But it's admittedly a big lack of feature in the current Python situation, yes. But it's also not a question of "CPAN", is it? CPAN doesn't uninstall things. It's a package repository. In any case this is a well known problem being looked at and in the process of being solved. > ?- etc... Up until this, you wore concrete and could explain exactly what you didn't like. You are partly incorrect and for some reason very angry, which is nonconstructive but beside the point. Your claim that it's not of features and concrete problems but a "state of mind" was countered by your concrete answers. ;-) Now if you cool down a bit we probably can find the actual problems. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Dec 22 09:02:28 2009 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 22 Dec 2009 09:02:28 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> Message-ID: <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> On Tue, Dec 22, 2009 at 02:07, David Cournapeau wrote: > But there is much more than what PEP 314 defines. Think about > everything which is in an egg, for example. How are files locations > are described, how are files outside site-packages described, etc... What do you mean "how"? > As a concrete example, there is not enough metadata in every installer > so that you can convert from one to the other, even though this would > be very useful and definitely possible. This I don't understand. What do you mean with "installer"? What kind of conversions do you refer to? > For example, you can upload a distribution which does not even have a > name, or a version, easy_install tries to find tarballs by scrapping > webpages, etc... Well, if you like your distribution to be called UNKNOWN-0.0, then I guess you can. But you also get warnings like this: warning: sdist: standard file not found: should have one of README, README.txt warning: sdist: missing required meta-data: name, url warning: sdist: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied In my opinion that's your problem of you ignore those, but maybe I'm wrong. Convince me. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cournape at gmail.com Tue Dec 22 09:11:38 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 17:11:38 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> Message-ID: <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> 2009/12/22 Lennart Regebro : > On Tue, Dec 22, 2009 at 02:07, David Cournapeau wrote: >> But there is much more than what PEP 314 defines. Think about >> everything which is in an egg, for example. How are files locations >> are described, how are files outside site-packages described, etc... > > What do you mean "how"? This is answered in my another email (build manifest and eggs). > >> As a concrete example, there is not enough metadata in every installer >> so that you can convert from one to the other, even though this would >> be very useful and definitely possible. > > This I don't understand. What do you mean with "installer"? What kind > of conversions do you refer to? By installer, I mean things produced by bdist_*. A significant portion of windows users don't like eggs, and prefer .exe-based (or .msi-based) installers. Currently, it is not possible to (reliably) convert from one to the other (e.g. eggs->wininst), but there is no reason why this is so. > >> For example, you can upload a distribution which does not even have a >> name, or a version, easy_install tries to find tarballs by scrapping >> webpages, etc... > > Well, if you like your distribution to be called UNKNOWN-0.0, then I > guess you can. But you also get warnings like this: > > warning: sdist: standard file not found: should have one of README, README.txt > warning: sdist: missing required meta-data: name, url > warning: sdist: missing meta-data: either (author and author_email) or > (maintainer and maintainer_email) must be supplied > > In my opinion that's your problem of you ignore those You are not wrong, but we are not talking about the same scenario. The scenario I care the most is user A build or install a package, and user B wants to install it; A may know very little about python. David From smueller at cpan.org Tue Dec 22 10:14:38 2009 From: smueller at cpan.org (Steffen Mueller) Date: Tue, 22 Dec 2009 09:14:38 +0000 (UTC) Subject: [Distutils] =?utf-8?q?Python_people_want_CPAN_and_how_the_latter_?= =?utf-8?q?came=09about?= References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> Message-ID: Hi Lennart, Lennart Regebro gmail.com> writes: > Interesting. I generally don't find automatically generated > documentation useful, but if it works for PERL I guess that's a > personal taste. Let me add two nits here: It's Perl, not PERL. The name of the language is *not* an acronym. Some people are really picky about that. More importantly, it's not "auto-generated" documentation per se. Not something like using doxygen to generate an API reference purely from the function/method name and signature. Instead, search.cpan.org (and kobesearch) extract the inlined documentation from the Perl modules. This documentation was written by the authors. It was *not* generated from the code itself. Here's what that looks like for a random one of my modules: http://search.cpan.org/~smueller/PAR-Repository-Client/lib/PAR/Repository/ Client.pm You could get the same page via http://search.cpan.org/perldoc?PAR::Repository::Client. This is an example of how CPAN works on namespace and distribution level. After a new file is uploaded, its contained namespaces (packages/class names) are added to the index if they're not already there. The index always contains a reference to the distribution that contains the newest *authorized* version of any namespace. Thus, you can access the distribution directly like in the first URL or have search.cpan.org go from package name to the most recent authorized version automatically (via the perldoc link). > > 4. Namespaces and some way of reserving them. There are likely many > > modules named postgresql on PyPI, but there's only one DBD::Pg (although > > there are other PostgreSQL modules that implement the perl DBI driver > > interface). This also helps with specifying dependencies. > > There is no reserving in the Python world, that's true. I'm skeptical. > There's enough problems with packages not being updated, we don't want > that for namespaces too, that would be terrible. We have a band of PAUSE admins with super cow powers to save the day. It's not like there's a lot of us, but so far, we've been able to handle authorization requests reasonably. Our policy in case an author goes missing who has permissions for a certain namespace are documented here: http://search.cpan.org/CPAN/modules/04pause.html#takeover > > It definitely appears to have the framework, but lacks some finishing > > touches that would enormously enhance usability. > > So far we have identified a "show all versions" link, and > automatically generated documentation. Thanks! Finally something > concrete! In a very concrete context: I really like how search.cpan.org will show you the latest stable version by default and provide drop-downs to select any developer (alpha) or old releases. It also has nice, consistent URLs as demonstrated above. ...~author will get me to author's overview page, .../dist/XXX will get me to the XXX distribution's overview, etc. Accessing any files within each distribution is possible via similar URLs. Another point that I really like about the service is that the distribution pages provide links to many other related services that are run by other volunteers. Take for example http://search.cpan.org/dist/PAR-Repository-Client/ There is a link to cpanforum, specifically the relevant discussion forum for the module at hand. It shows a link to the bug/request tracker with the number of open bugs, the one next to it will display a nice hierarchical (recursive) list of dependencies. Again. Run by another volunteer. The line below is dedicated to CPAN testers results. It shows the number of FAIL/PASS/UNKNOWN results followed by a link to the page with the raw reports and -- this is extremely useful as an author -- a colourful matrix that shows the test results depending on perl version and operating system. My example in this mail is not a very widely used module. Thus, the matrix is somewhat sparse, too: http://matrix.cpantesters.org/?dist=PAR-Repository-Client+0.25. But it clearly shows my module isn't compatible to really old versions of Perl (5.6). In some other cases, I might learn that it isn't working well on some operating system because its column is red or yellow. Here's an example of a more widely used and installed distribution: http://matrix.cpantesters.org/?dist=Data-Dumper+2.125. As an author, these tools help me identify weak points or portability problems in my software. As a user, this can help me understand whether I really want to rely on this piece of software for my business. Everyone wins. Please don't take any of this the wrong way. I'm not trying to say "look what we can do and you don't" at all. Not only do I have no idea what the python tools look like, but in the previous two paragraphs, I am really just trying to point out a few technical strong points of various CPAN-related services that are very valuable to me as a user, developer and contributor. Best regards, Steffen From jari.pennanen at gmail.com Tue Dec 22 10:49:22 2009 From: jari.pennanen at gmail.com (Jari Pennanen) Date: Tue, 22 Dec 2009 11:49:22 +0200 Subject: [Distutils] Swig build_py ran before build_ext In-Reply-To: <5b8d13220912211804s1752092bk2518ee616dfdc211@mail.gmail.com> References: <6d0569fc0912211416p63c8e106na3fafe80cbf22589@mail.gmail.com> <5b8d13220912211804s1752092bk2518ee616dfdc211@mail.gmail.com> Message-ID: <6d0569fc0912220149t1227eda1h7a3a8dc7fd55024c@mail.gmail.com> Well, if that is so, I will paste my workaround for the sake of completeness if other people come here to wonder: #!/usr/bin/env python from distutils.core import setup, Extension from distutils.command.build_py import build_py dist = setup(...) # Rerun the build_py to ensure that swig generated py files are also copied build_py = build_py(dist) build_py.ensure_finalized() build_py.run() Btw, this could be mentioned in http://docs.python.org/distutils/setupscript.html since SWIG is rather common tool... it took me a notable time to figure this out. (Sorry about duplicate mail) 2009/12/22 David Cournapeau : > On Tue, Dec 22, 2009 at 7:16 AM, Jari Pennanen wrote: >> Hi! >> >> I have old/new problem, once mentioned in 2004 at this same mailing >> list ( http://mail.python.org/pipermail/distutils-sig/2004-March/003807.html >> ) but surely that implementation is bad... But the problem persists, I >> use SWIG and had to Google all over to only find out build order >> cannot be changed. >> >> Someone should implement build_order keyword argument to setup, e.g.: >> >> ? ?setup(..., build_order=['build_ext', 'build_py']) >> >> If some orders are missing, like in above example: build_clib, >> build_scripts, they would be appended as last. Anyways, the point is >> make the build order customizable through setup keyword. > > This would need to be tested/confirmed on real packages, but I am > afraid it would break a lot of packages, especially the ones extending > distutils. > > cheers, > > David > -- Tmi: Jari Pennanen Y-tunnus: 1867270-2 Puh: 050 4911400 From ziade.tarek at gmail.com Tue Dec 22 11:19:46 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 22 Dec 2009 11:19:46 +0100 Subject: [Distutils] Swig build_py ran before build_ext In-Reply-To: <6d0569fc0912220149t1227eda1h7a3a8dc7fd55024c@mail.gmail.com> References: <6d0569fc0912211416p63c8e106na3fafe80cbf22589@mail.gmail.com> <5b8d13220912211804s1752092bk2518ee616dfdc211@mail.gmail.com> <6d0569fc0912220149t1227eda1h7a3a8dc7fd55024c@mail.gmail.com> Message-ID: <94bdd2610912220219v61ce1481k8b282e584dc04b46@mail.gmail.com> On Tue, Dec 22, 2009 at 10:49 AM, Jari Pennanen wrote: > Well, if that is so, I will paste my workaround for the sake of > completeness if other people come here to wonder: > > ? #!/usr/bin/env python > ? from distutils.core import setup, Extension > ? from distutils.command.build_py import build_py > > ? dist = setup(...) > > ? # Rerun the build_py to ensure that swig generated py files are also copied > ? build_py = build_py(dist) > ? build_py.ensure_finalized() > ? build_py.run() > > Btw, this could be mentioned in > http://docs.python.org/distutils/setupscript.html since SWIG is rather > common tool... it took me a notable time to figure this out. Having an option to define explicitely a custom order for the subcommands for the build command in setup.py sounds like a feature we could add. Please add an issue in bugs.python.org, I'll add Marc-Andr? in cc. there and we can investigate in this, Regards Tarek From cournape at gmail.com Tue Dec 22 14:02:12 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Dec 2009 22:02:12 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> Message-ID: <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> On Tue, Dec 22, 2009 at 4:43 PM, Lennart Regebro wrote: > On Mon, Dec 21, 2009 at 23:48, David Cournapeau wrote: >> mind". Reliable packaging requires explicit handling, where the whole >> python stack for packaging relies a lot on implicit behavior. > > Can you give examples of this implicit behaviour? >From the top of my head: - how data files are included - how sdist is produced (a lot of sdist-generated tarballs contain lot of junk like *.pyc or egg-info). I think we already had this discussion, and that I was the only one who thought this was a problem. - how files are installed: distutils just copy whatever is in the build directory for at least some files, so you cannot easily check which files are installed - hence my proposal on a build manifest to make this step explicit. This is maybe the number one issue we have in numpy/scipy where people unfamiliar with distutils warts try to build/install numpy/scipy. > It's not worse, it's the same.But it's admittedly a big lack of > feature in the current Python situation, yes. But it's also not a > question of "CPAN", is it? CPAN doesn't uninstall things. Yes, CPAN does not uninstall things, which is why I was making the difference betwen explicitely uninstally things and (implicitely) uninstalling beforere reinstalling things. The former would not be done though CPAN, the later would. > > Up until this, you wore concrete and could explain exactly what you > didn't like. You are partly incorrect and for some reason very angry, > which is nonconstructive but beside the point. Your claim that it's > not of features and concrete problems but a "state of mind" was > countered by your concrete answers. ;-) Features only help as much as long as you agree on the purpose (or "state of mind" as I put it). A lot of things I consider misfeatures in the whole packaging ecosystem in python are justified by being convenience to the developer, at the cost of reliability (at least in my opinion). Data files is my favorite example, as it is deceptively simple feature-wise, and yet so confusing. We had 2 emails in the recent days on this ML on the topic, and even people very familiar with distutils got it wrong. Just considering numpy.distutils/distutils/setuptools, there are 5 ways to handle data files at least (data_files, package_data, MANIFEST[.in], VCS, add_data_dir). None of them makes it easy to install a data file in arbitrary location. I think distutils/setuptools lacks a robust way of including data files, but it is not as much a feature problem (which could be solved relatively easily), as more a problem of what people want. The devil is really in the details for packaging, which is why explicit is a very desirable feature. Someone noticed earlier that distutils was the exact contrary of the python zen on some core issues: more than one way to do it, lack of simplicity, often implicit. I think this describe the problem quite well, and I am deeply convinced that that's the root cause of the failure of Pypi for a subset of the community. David From ziade.tarek at gmail.com Tue Dec 22 14:35:15 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 22 Dec 2009 14:35:15 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> Message-ID: <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> On Tue, Dec 22, 2009 at 2:02 PM, David Cournapeau wrote: [..] > > Data files is my favorite example, as it is deceptively simple > feature-wise, and yet so confusing. We had 2 emails in the recent days > on this ML on the topic, and even people very familiar with distutils > got it wrong. Just considering numpy.distutils/distutils/setuptools, > there are 5 ways to handle data files at least (data_files, > package_data, MANIFEST[.in], VCS, add_data_dir). None of them makes it > easy to install a data file in arbitrary location. I think > distutils/setuptools lacks a robust way of including data files, but > it is not as much a feature problem (which could be solved relatively > easily), as more a problem of what people want. I'll speak about distutils only here. That topic is a good example indeed. See: http://bugs.python.org/issue2279 1/ Someone added an issue tracker in bugs.python.org because those data files are not added unless you add an explicit MANIFEST.in 2/ I worked to change that 3/ it is now included in Python 2.7 So now I am asking you: what do *you* want ? When you say "which could be solved relatively easily" I suggest that you take the time to add concise and precise proposals in bugs.python.org so I can work on them. And when you say "even people very familiar with distutils got it wrong" I suggest that you also take the time to propose some doc fixes, or points its lacking in the tracker. Distutils has now a maintainer, and that would be me. In two days, it will be one year exactly since I've taken over the maintenance and I've done so far 185 commits in the distutils package, and 15 commits in its documentation and this rate is going to increase once the PEPs we are building are accepted. Unfortunately I can't make it a great package in just one year, and there is a huge amount of work left to be done. So I suggest that you take the chance to help me fixing the stuff you don't like in that package by feeding the tracker. Patches welcome ! ;) Cheers, Tarek -- Tarek Ziad? | http://ziade.org From jari.pennanen at gmail.com Tue Dec 22 15:36:52 2009 From: jari.pennanen at gmail.com (Jari Pennanen) Date: Tue, 22 Dec 2009 16:36:52 +0200 Subject: [Distutils] Swig build_py ran before build_ext In-Reply-To: <94bdd2610912220219v61ce1481k8b282e584dc04b46@mail.gmail.com> References: <6d0569fc0912211416p63c8e106na3fafe80cbf22589@mail.gmail.com> <5b8d13220912211804s1752092bk2518ee616dfdc211@mail.gmail.com> <6d0569fc0912220149t1227eda1h7a3a8dc7fd55024c@mail.gmail.com> <94bdd2610912220219v61ce1481k8b282e584dc04b46@mail.gmail.com> Message-ID: <6d0569fc0912220636i69dd671bw41c67ed68605674a@mail.gmail.com> Here is feature request: http://bugs.python.org/issue7562 2009/12/22 Tarek Ziad? : > On Tue, Dec 22, 2009 at 10:49 AM, Jari Pennanen wrote: >> Well, if that is so, I will paste my workaround for the sake of >> completeness if other people come here to wonder: >> >> ? #!/usr/bin/env python >> ? from distutils.core import setup, Extension >> ? from distutils.command.build_py import build_py >> >> ? dist = setup(...) >> >> ? # Rerun the build_py to ensure that swig generated py files are also copied >> ? build_py = build_py(dist) >> ? build_py.ensure_finalized() >> ? build_py.run() >> >> Btw, this could be mentioned in >> http://docs.python.org/distutils/setupscript.html since SWIG is rather >> common tool... it took me a notable time to figure this out. > > Having an option to define explicitely a custom order for the > subcommands for the build command in setup.py sounds like a feature we > could add. > > Please add an issue in bugs.python.org, I'll add Marc-Andr? in cc. > there and we can investigate in this, > > Regards > Tarek > -- Tmi: Jari Pennanen Y-tunnus: 1867270-2 Puh: 050 4911400 From regebro at gmail.com Tue Dec 22 19:04:00 2009 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 22 Dec 2009 19:04:00 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> Message-ID: <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> 2009/12/22 David Cournapeau : > By installer, I mean things produced by bdist_*. A significant portion > of windows users don't like eggs, and prefer .exe-based (or > .msi-based) installers. Currently, it is not possible to (reliably) > convert from one to the other (e.g. eggs->wininst), but there is no > reason why this is so. You are right, I can't see any reason why not. Why can't you? Obviously you can do it from a source distribution, but what information is missing from the binary distributions? > You are not wrong, but we are not talking about the same scenario. The > scenario I care the most is user A build or install a package, and > user B wants to install it; A may know very little about python. In which case no sane user B wants to install it. ;) How is this a different scenario? What scenario was I talking about? On Tue, Dec 22, 2009 at 14:02, David Cournapeau wrote: > From the top of my head: > - how data files are included Well, there are rules for that, just as with python files, you dont' have to explicitly specify every file that should be included, thank god, that would be really annoying. And this has WHAT to do with CPAN? Just to try to keep even remotely on topic? Isn't this turning into the usual "distutils sucks and should be rewritten but I can't explain why and how" rant? And isn't the answer still: "Well, do it then?" > Features only help as much as long as you agree on the purpose (or > "state of mind" as I put it). A lot of things I consider misfeatures > in the whole packaging ecosystem in python are justified by being > convenience to the developer, at the cost of reliability (at least in > my opinion). > > Data files is my favorite example, as it is deceptively simple > feature-wise, and yet so confusing. We had 2 emails in the recent days > on this ML on the topic, and even people very familiar with distutils > got it wrong. Just considering numpy.distutils/distutils/setuptools, > there are 5 ways to handle data files at least (data_files, > package_data, MANIFEST[.in], VCS, add_data_dir). None of them makes it > easy to install a data file in arbitrary location. I think > distutils/setuptools lacks a robust way of including data files, but > it is not as much a feature problem (which could be solved relatively > easily), as more a problem of what people want. > > The devil is really in the details for packaging, which is why > explicit is a very desirable feature. Someone noticed earlier that > distutils was the exact contrary of the python zen on some core > issues: more than one way to do it, lack of simplicity, often > implicit. I think this describe the problem quite well, and I am > deeply convinced that that's the root cause of the failure of Pypi for > a subset of the community. Well, you have completely failed in explaining why CPAN is fantastic and PyPI sucks, probably because your answer isn't about CPAN at all, but just the same old packaging rant. It really grows old you know. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue Dec 22 19:15:30 2009 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 22 Dec 2009 19:15:30 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> Message-ID: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> On Tue, Dec 22, 2009 at 10:14, Steffen Mueller wrote: > Hi Lennart, > > Lennart Regebro gmail.com> writes: >> Interesting. I generally don't find automatically generated >> documentation useful, but if it works for PERL I guess that's a >> personal taste. > > Let me add two nits here: > It's Perl, not PERL. The name of the language is *not* an acronym. Some people > are really picky about that. In a language where you can spell any functionality in millions of ways, you are not allowed to spell the language however you want!? ;-) Ok, P?rl. I mean Perl. :-) > More importantly, it's not "auto-generated" documentation per se. Not something > like using doxygen to generate an API reference purely from the function/method > name and signature. Instead, search.cpan.org (and kobesearch) extract the > inlined documentation from the Perl modules. This documentation was written by > the authors. It was *not* generated from the code itself. Here's what that looks > like for a random one of my modules: > http://search.cpan.org/~smueller/PAR-Repository-Client/lib/PAR/Repository/ > Client.pm OK, so that would be docstrings and stuff then. It's true, if we had something like that it would work as an incentive for making more such documentation. > You could get the same page via > http://search.cpan.org/perldoc?PAR::Repository::Client. This is an example of > how CPAN works on namespace and distribution level. After a new file is > uploaded, its contained namespaces (packages/class names) are added to the > index if they're not already there. How does it handle if two modules implement the same namespace? > The index always contains a reference to > the distribution that contains the newest *authorized* version of any > namespace. Who authorizes it? > We have a band of PAUSE admins with super cow powers to save the day. It's not > like there's a lot of us, but so far, we've been able to handle authorization > requests reasonably. Our policy in case an author goes missing who has > permissions for a certain namespace are documented here: > http://search.cpan.org/CPAN/modules/04pause.html#takeover I have to say that I vastly prefer not to have any authorization and allow anyone to release anything in any namespace. But then I am getting fanatically anarchical in these issues. You can not organize freedom. >> So far we have identified a "show all versions" link, and >> automatically generated documentation. Thanks! Finally something >> concrete! > > In a very concrete context: I really like how search.cpan.org will show you the > latest stable version by default and provide drop-downs to select any developer > (alpha) or old releases. It also has nice, consistent URLs as demonstrated > above. ...~author will get me to author's overview page, .../dist/XXX will get > me to the XXX distribution's overview, etc. Accessing any files within each > distribution is possible via similar URLs. The only thing I can see there that's missing is the "drop down", or link to a list of all versions or similar. That would also make it easier to make alpha versions, they would be easily viewable, but not shown and listed by default. We should suggest that to Martin v L?wis. > Another point that I really like about the service is that the distribution > pages provide links to many other related services that are run by other > volunteers. Take for example http://search.cpan.org/dist/PAR-Repository-Client/ > There is a link to cpanforum, specifically the relevant discussion forum for the > module at hand. It shows a link to the bug/request tracker with the number of > open bugs, the one next to it will display a nice hierarchical (recursive) list > of dependencies. Right, those third-party services doesn't exist for PyPI. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue Dec 22 19:48:38 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Dec 2009 19:48:38 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> Message-ID: <4B311486.9030006@v.loewis.de> > By installer, I mean things produced by bdist_*. A significant portion > of windows users don't like eggs, and prefer .exe-based (or > .msi-based) installers. Currently, it is not possible to (reliably) > convert from one to the other (e.g. eggs->wininst), but there is no > reason why this is so. There is most certainly a reason. The binary distribution may be lacking pieces of source code that would be needed to build another (different) binary distribution. For example, if you have a Linux RPM or .deb, it is, in general, not possible to create a Windows installer out of this, as you may need to recompile extension modules. Including the full source code in the binary distribution just to support this rare use case is unreasonable: people who want a different package format can build from the source package. Regards, Martin From robert.kern at gmail.com Tue Dec 22 19:54:42 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 22 Dec 2009 12:54:42 -0600 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B311486.9030006@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <4B311486.9030006@v.loewis.de> Message-ID: On 2009-12-22 12:48 PM, "Martin v. L?wis" wrote: >> By installer, I mean things produced by bdist_*. A significant portion >> of windows users don't like eggs, and prefer .exe-based (or >> .msi-based) installers. Currently, it is not possible to (reliably) >> convert from one to the other (e.g. eggs->wininst), but there is no >> reason why this is so. > > There is most certainly a reason. The binary distribution may be lacking > pieces of source code that would be needed to build another (different) > binary distribution. > > For example, if you have a Linux RPM or .deb, it is, in general, not > possible to create a Windows installer out of this, as you may need > to recompile extension modules. Including the full source code in the > binary distribution just to support this rare use case is unreasonable: > people who want a different package format can build from the source > package. Yes, that use case is rare (and bizarre), but the desire to convert Windows-platform eggs to .msi or .exe installers is not so rare and does not present the same level of difficulty. The latter is the use case David was referring to, not the one you outlined. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From martin at v.loewis.de Tue Dec 22 20:20:17 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 22 Dec 2009 20:20:17 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <4B311486.9030006@v.loewis.de> Message-ID: <4B311BF1.90001@v.loewis.de> > Yes, that use case is rare (and bizarre), but the desire to convert > Windows-platform eggs to .msi or .exe installers is not so rare and does > not present the same level of difficulty. The latter is the use case > David was referring to, not the one you outlined. Why do people want to do that? Can't they build the type of package they want from the source package? Regards, Martin From robert.kern at gmail.com Tue Dec 22 20:24:09 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 22 Dec 2009 13:24:09 -0600 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B311BF1.90001@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <4B311486.9030006@v.loewis.de> <4B311BF1.90001@v.loewis.de> Message-ID: On 2009-12-22 13:20 PM, "Martin v. L?wis" wrote: >> Yes, that use case is rare (and bizarre), but the desire to convert >> Windows-platform eggs to .msi or .exe installers is not so rare and does >> not present the same level of difficulty. The latter is the use case >> David was referring to, not the one you outlined. > > Why do people want to do that? Can't they build the type of package they > want from the source package? Not necessarily on Windows. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david.lyon at preisshare.net Tue Dec 22 23:02:35 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 22 Dec 2009 17:02:35 -0500 Subject: [Distutils] =?utf-8?q?Python_people_want_CPAN_and_how_the_latter_?= =?utf-8?q?came=09about?= In-Reply-To: <4B311486.9030006@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <4B311486.9030006@v.loewis.de> Message-ID: <18680da20f48dbdd65c18bdf2268096e@preisshare.net> On Tue, 22 Dec 2009 19:48:38 +0100, "Martin v. L?wis" wrote: >> .. > > There is most certainly a reason. The binary distribution may be lacking > pieces of source code that would be needed to build another (different) > binary distribution. Actually, I have a question about that. I'll just ask here because you may be able to answer it in 30 seconds but it might take me a few hours to find out the answer myself. Under Windows, through the build-msi and build-exe procedure, we have windows GUI package installers. When I was using one built by Mark Hammond the other day, I noticed that it worked pretty fabulously. However, under Windows 7 Enterprise, with a high security policy. It's totally impossible now to install the same package, even if python has been installed by the system administrator. It won't let you run .exe files. Since this whole approach now seems broken going forward.. what is going to happen? Can that code be moved somewhere so that it can install regular (source) packages? David From david.lyon at preisshare.net Tue Dec 22 23:34:07 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 22 Dec 2009 17:34:07 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> Message-ID: On Tue, 22 Dec 2009 19:04:00 +0100, Lennart Regebro wrote: > 2009/12/22 David Cournapeau : >> >> .. >> >> The devil is really in the details for packaging, which is why >> explicit is a very desirable feature. Someone noticed earlier that >> distutils was the exact contrary of the python zen on some core >> issues: more than one way to do it, lack of simplicity, often >> implicit. I think this describe the problem quite well, and I am >> deeply convinced that that's the root cause of the failure of Pypi for >> a subset of the community. > > Well, you have completely failed in explaining why CPAN is fantastic > and PyPI sucks, probably because your answer isn't about CPAN at all, > but just the same old packaging rant. It really grows old you know. Oh come on... nobody is saying pypi sucks. Only that the package installation experience in python does. In python, these are two separate and entirely different things. Under Perl, the whole thing is just called CPAN. There's one CPAN client that you use (btw - that gets included in your perl distribution) and it works. Guido is being harassed about by users from the scientific community as I understand it. They are after a 'nice' user experience like they get in Perl. So let me translate as best as I can ... "Python people want _NICE_USER_EXPERIENCE_ and how the latter came about" David From regebro at gmail.com Wed Dec 23 01:01:11 2009 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 23 Dec 2009 01:01:11 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> Message-ID: <319e029f0912221601o4dac1df4kff1414fef26ec3dc@mail.gmail.com> On Tue, Dec 22, 2009 at 23:34, David Lyon wrote: > Oh come on... nobody is saying pypi sucks. Only that the package > installation experience in python does. In python, these are two > separate and entirely different things. > > Under Perl, the whole thing is just called CPAN. OK, so in the Perl community there is apparently a lot of confusion on what CPAN is. And in the Python community there is a lot of confusion about installation. And although CPAN does't install and has nothing to do with installation, people in the Perl community think it does. So according to you, when Perl people say "Python need CPAN" what they actually mean, according to you, is "We can't convert one package format to another", because they don't know that CPAN is the archive, not the installation program. Is this correct? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Wed Dec 23 02:58:10 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 23 Dec 2009 02:58:10 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <18680da20f48dbdd65c18bdf2268096e@preisshare.net> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <4B311486.9030006@v.loewis.de> <18680da20f48dbdd65c18bdf2268096e@preisshare.net> Message-ID: <4B317932.4060206@v.loewis.de> > However, under Windows 7 Enterprise, with a high security policy. It's > totally impossible now to install the same package, even if python has > been installed by the system administrator. It won't let you run .exe > files. I don't think that's the case - Windows 7 most certainly lets you run exe files. IOW, I don't know what the problem might be that you are experiencing. Regards, Martin From david.lyon at preisshare.net Wed Dec 23 02:56:13 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 22 Dec 2009 20:56:13 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912221601o4dac1df4kff1414fef26ec3dc@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> <319e029f0912221601o4dac1df4kff1414fef26ec3dc@mail.gmail.com> Message-ID: <51c331ec3db84cceb8f2a8cf911a9aac@preisshare.net> On Wed, 23 Dec 2009 01:01:11 +0100, Lennart Regebro wrote: > OK, so in the Perl community there is apparently a lot of confusion on > what CPAN is. CPAN is plain and simple. There is no confusion, because there is just one 'brand-name' for the whole kit and caboodle. "Packaging" in the perl world just goes under the name CPAN. > And in the Python community there is a lot of confusion > about installation. Relatively speaking, yes. > And although CPAN does't install and has nothing > to do with installation, people in the Perl community think it does. That's right. Because of the bundling. > So according to you, when Perl people say "Python need CPAN" what they > actually mean, according to you, is "We can't convert one package > format to another", because they don't know that CPAN is the archive, > not the installation program. > > Is this correct? The question is a little too complicated to answer directly on that wording. However, a Perl user (speaking for myself) would typically know nothing about conversion of different package formats, because as far as I am aware the user isn't exposed to that level of complexity. There is no .tar.gz, .zip, .bz2, .exe, .msi or .egg concept of packages in perl. And having to pick one.. that may or may not be right for your configuration. Call a perl user a pampered pooch by all means. But all they know is that if they need a module.. then they use CPAN. Really, some of these questions should go back up the management chain for answering. We shouldn't be having expectations that we can write code when drunk, fix it in the morning, and be merry and listen to ump-pah-pah music (like CPAN people). Especially if the development process is to be reading and commenting on PEPs. Please.. lets just go back to the work at hand.. This is serious.. David From david.lyon at preisshare.net Wed Dec 23 03:14:13 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 22 Dec 2009 21:14:13 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B317932.4060206@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <4B311486.9030006@v.loewis.de> <18680da20f48dbdd65c18bdf2268096e@preisshare.net> <4B317932.4060206@v.loewis.de> Message-ID: <2ef76e108dd688b50400fac5a3b6cfa3@preisshare.net> On Wed, 23 Dec 2009 02:58:10 +0100, "Martin v. L?wis" wrote: >> However, under Windows 7 Enterprise, with a high security policy. It's >> totally impossible now to install the same package, even if python has >> been installed by the system administrator. It won't let you run .exe >> files. > > I don't think that's the case - Windows 7 most certainly lets you run > exe files. > > IOW, I don't know what the problem might be that you are experiencing. Sorry for the noise. It's something to do with the new security policy that I don't understand. It won't run any program but the installed programs. Thanks for asking - Merry Christmas David From cournape at gmail.com Wed Dec 23 04:39:25 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 23 Dec 2009 12:39:25 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> Message-ID: <5b8d13220912221939q55db661bgb29e45d05aada746@mail.gmail.com> 2009/12/23 Lennart Regebro : > > ?And this has WHAT to do with CPAN? Just to try to keep even remotely > on topic? The fact that you consider it is not linked is exactly what's wrong in my own opinion. If you want a reliable distribution channel, its content (the installers) has to be reliable, and for the content to be reliable, the tools to produce them have too as well. When you have a failure with easy_install about some sandbox violation, missing file, etc... it is a failure of Pypi for the user, but the underlying reason is the packaging tool in the first place. > And isn't > the answer still: "Well, do it then?" http://github.com/cournape/toydist.git which is why I am trying to get some agreement on at least some low level mechanisms which can be shared between tools. The exact thing you already complain last time this was discussed. David From cournape at gmail.com Wed Dec 23 05:12:30 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 23 Dec 2009 13:12:30 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> Message-ID: <5b8d13220912222012p69cdc40arcf262c7c02e0dc7d@mail.gmail.com> On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziad? wrote: > > When you say "which could be solved relatively easily" I suggest that > you take the time to add concise and precise proposals in > bugs.python.org so I can work on them. Technically, it is easy. Only have two mechanisms for data files: one for installed data files, and one for extra source files (as done in automake for example): - Extra files only need to be listed (and included in sdist) - Install data files need a target directory. One of the problem with distutils here is that you can only hardcode paths - in my own packaging solution, I use $path variables so that any path defined internally can be reused ($bindir, etc...); something similar could be defined in distutils. The remaining issue is then how to resolve the final target: installing src/foo.dat into $datadir installs $datadir/src/foo.dat or $datadir/foo.dat ? In my own package, I use a source directory argument, to reproduce the nobase scheme of automake: http://sources.redhat.com/automake/automake.html#Basics-of-Installation The way I did it is to have: target=/usr/include source=include files=foo.h,sys/bar.h The files include/foo.h is installed as /usr/include/foo.h and include/sys/bar.h is installed as usr/include/sys/bar.h (the full path in the file sections is appended to target) If you want to install bar.h as /usr/include/bar.h, you would need two sections: target=/usr/include source=include files=foo.h And target=/usr/include source=include/sys files=bar.h Assuming you can use $path in target, this can describe any scheme I can think of, and I don't think it can be made simpler (I would be happy to know of a simpler solution though). cheers, David From david.lyon at preisshare.net Wed Dec 23 07:37:44 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 23 Dec 2009 01:37:44 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> Message-ID: <986a171837816a1ce3c8eb9709c45139@preisshare.net> On Tue, 22 Dec 2009 19:04:00 +0100, Lennart Regebro wrote: > Isn't this turning into the usual "distutils sucks and > should be rewritten but I can't explain why and how" rant? Len, I just want to revisit this, to hopefully close it. Those words are entirely yours. Not those who are able to explain how and why, with or without the rant. David C sums it up very nicely by saying that distutils works by implicit specification of build information. That's in contrast to the rest of the software world - using explicit build systems. If anything, everybody who's saying anything on the list now understands this and can live with it. For better or worse. There's some progress underway. There are commits happening. One very important thing to remember is just who started this request to look into 'cpan' style packaging in the first place. If I'm not wrong, it came right from the top. So go hassle whoever that might be. But even then, that's just a response to calls by plain ordinary users in the field. With CPAN, they would never label discussion or feedback as 'rant'. Actually, they went and had a beer, and fixed most of the issues *before* there could even be a complaint. That's still true today. > And isn't the answer still: "Well, do it then?" I refer you to reread Steffens original post. It is very helpful. No one person is able to do this. That being, redo distutils. There's something in Steffens post about 'the sum of the parts being greater..'. That entirely sums it up for me. It's too big a job for any one person to commence. One has to question what the return on investment would be. One could estimate that even a cheap distutils replacement could cost $80,000. That might get something - i guess. But who has that sort of time/money to throw around ? Certainly not the banks or the stock exchange brokers who use python for their own commercial benefit. So why ask it from the volunteers ? And if python management aren't united on the need for it, doing a distutils replacement becomes an even weaker proposition. If there is work underway.. lets just get it finished.. PEP-345 has been open since 2005. In CPAN it just would have been a beer fight and have been over in a weekend. Whatever... David From regebro at gmail.com Wed Dec 23 10:08:24 2009 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 23 Dec 2009 10:08:24 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <51c331ec3db84cceb8f2a8cf911a9aac@preisshare.net> References: <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> <319e029f0912221601o4dac1df4kff1414fef26ec3dc@mail.gmail.com> <51c331ec3db84cceb8f2a8cf911a9aac@preisshare.net> Message-ID: <319e029f0912230108q274ab3cbr9015d2ac4cbbd121@mail.gmail.com> On Wed, Dec 23, 2009 at 02:56, David Lyon wrote: > On Wed, 23 Dec 2009 01:01:11 +0100, Lennart Regebro > wrote: >> OK, so in the Perl community there is apparently a lot of confusion on >> what CPAN is. > > CPAN is plain and simple. There is no confusion, because there is > just one 'brand-name' for the whole kit and caboodle. "Packaging" > in the perl world just goes under the name CPAN. That's not what Steffen Mueller said. > However, a Perl user (speaking for myself) would typically > know nothing about conversion of different package formats, > because as far as I am aware the user isn't exposed to that level > of complexity. > > There is no .tar.gz, .zip, .bz2, .exe, .msi or .egg concept of > packages in perl. And having to pick one.. that may or may not > be right for your configuration. So your usecase, that a Windows user refuses to install anything else than .msi files is solved in CPAN by the user not installing using CPAN. Well, that's handy. And the Python situation is worse how, exactly? > Call a perl user a pampered pooch by all means. But all they > know is that if they need a module.. then they use CPAN. Right, and in Python neither easy_installer nor pip is included by default. That's obviously a big plus for Perl. After that your argumentation becomes self-contradictory and confused and reverts in to the usual distutils rant, which still isn't constructive. I'd also like an installer to be included in Python core, but there are very good reasons for only including things that are stable and almost unanimously accepted. And although pip looks like it's going to get unanimously accepted it hasn't been yet. And obviously a very vocal minority that constantly does nothing but waste their and others time by complaining on distutils but not doing anything to fix it or replace it isn't helping either. On Wed, Dec 23, 2009 at 07:37, David Lyon wrote: > Those words are entirely yours. Not those who are able > to explain how and why, with or without the rant. I do hope you are not including yourself amongst those who are able to explain how and why. > One very important thing to remember is just who started > this request to look into 'cpan' style packaging in the > first place. > > If I'm not wrong, it came right from the top. So go > hassle whoever that might be. But even then, that's just > a response to calls by plain ordinary users in the field. That is the most ridicolous excuse I've heard in a long time. People are saying to Guido that "Python needs CPAN" and wonders what that means. You come with you usual anti distutils rant, I ask *you* what the connection is, and you tell med to go hassle Guido. Pathetic. Honestly. > With CPAN, they would never label discussion or feedback > as 'rant'. No, but they would label rants as rants, and what you do is ranting. There is a bit of constructive feedback from you but there sure is no discussion. > I refer you to reread Steffens original post. It is very > helpful. I completely agree. Maybe you should read it and try to figure out WHY it was helpful, when your answers aren't? > No one person is able to do this. That being, redo distutils. You are not the only one who says it needs to be rewritten from scratch. If you are saying that the few people that exists that agree with you about this *also* is not enough, then you are saying that the community does not agree with you that it needs to be rewritten. > And if python management aren't united on the need for it, > doing a distutils replacement becomes an even weaker > proposition. So you are saying that you think distutils must be replaced, and that the only persons who understand why it needs to be replaced are not willing or able to do or even start the work. > PEP-345 has been open since 2005. In CPAN it just would > have been a beer fight and have been over in a weekend. Maybe they have less people who complain, and use up their and other peoples energies by ranting and complaining instead of coding. In any case, I take your answer as a definite statement from your side that you will not help neither fix nor replace distutils. And then I honestly see no reason why I should listen to you anymore. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Wed Dec 23 10:11:18 2009 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 23 Dec 2009 10:11:18 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912221939q55db661bgb29e45d05aada746@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> <5b8d13220912221939q55db661bgb29e45d05aada746@mail.gmail.com> Message-ID: <319e029f0912230111p6c1a86c1h7aee8edce30b9389@mail.gmail.com> 2009/12/23 David Cournapeau : > which is why I am trying to get some agreement on at least some low > level mechanisms which can be shared between tools. The exact thing > you already complain last time this was discussed. And then I also asked a question, which I have asked many times: Will a replacement of distutils follow the distribution/install PEPs that are being discussed and designed at the moment? But it's good to see a start on the effort you claim is necessary. Thank you. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ziade.tarek at gmail.com Wed Dec 23 11:28:23 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 11:28:23 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912222012p69cdc40arcf262c7c02e0dc7d@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> <5b8d13220912222012p69cdc40arcf262c7c02e0dc7d@mail.gmail.com> Message-ID: <94bdd2610912230228q2e75a18ak3f695d1bc8a7785e@mail.gmail.com> On Wed, Dec 23, 2009 at 5:12 AM, David Cournapeau wrote: > On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziad? wrote: > >> >> When you say "which could be solved relatively easily" I suggest that >> you take the time to add concise and precise proposals in >> bugs.python.org so I can work on them. > > Technically, it is easy. Only have two mechanisms for data files: one > for installed data files, and one for extra source files (as done in > automake for example): > ?- ?Extra files only need to be listed (and included in sdist) > ?- ?Install data files need a target directory. One of the problem > with distutils here is that you can only hardcode paths - in my own > packaging solution, I use $path variables so that any path defined > internally can be reused ($bindir, etc...); something similar could be > defined in distutils. distutils uses install schemes with variables too ($base, etc), that are expanded at installation time. and differs depending on the options you pass to the install command. (look at the command/install module) There's only one target directory for all files describe in data_file though, but we could work on extending this if you make a feature request in the bug tracker, From ziade.tarek at gmail.com Wed Dec 23 11:50:56 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 11:50:56 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <4B301D54.6050109@v.loewis.de> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> Message-ID: <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> On Tue, Dec 22, 2009 at 2:13 AM, "Martin v. L?wis" wrote: >> FYI. This Distutils-SIG thread is about proposing PEP 345 and impacts PyPI. >> So if there's anything that look suspicious to anyone, please join the >> discussion at Distutils-SIG > > In a number of places (Requires-Dist, Requires-Python), comma-separated > lists of version constraints are used. The PEP needs to specify what the > collective constraint means that gets specified. > > Most likely, the intended meaning is that the constraints get > and-combined, in which case the example > > Requires-Python: 2.5, 2.6 > > is non-sensical (no Python release meets that constraint). (else, > if it was meant to denote or-combination, then ">1.0, !=1.3.4, <2.0" > would be non-sensical, specifying no constrating at all) Right, this is not good because the comma means "and" then "or" I see two ways to fix this: - introduce groups, and say the the comma is "or" inside a group: 0.9, (>1.0, !=1.3.4, <2.0), 3.4 ==> 0.9 or (>1.0 and !=1.3.4 and <2.0) or 3.4 - use an explicit separator word (and/or) > > For the Copyright field, I'm not sure what the purpose of it is. > If it is what I think it is (listing attribution), it should probably > be specified as "multiple use". I just noticed this field was added after PEP 314. I don't think we should keep it at all. I don't see a use case for this (README etc.. are enough for copyright matters) > > For Documentation, I think the entire field must be reconsidered. > If it is really meant to be reStructuredText, then the spec should > explain how to do leading spaces/indentation. Yes, this is the case in fact. but the way it currently works is deficient because you may lose empty lines because the PKG-INFO file works. What I have suggested last week was to introduce a delimiter to start each new line so this field stays RFC 822 compatible and reST works if we want to read back the value: Description: This module collects votes from beagles | in order to determine their electoral wishes. | Do *not* try to use this module with basset hounds; | it makes them grumpy. | |?example code : | | >>> 2 + 1 | 3 A cleaner option of course, is to drop the RFC 822 format and to use a simpler format like json for example, since we have a reader/writer for that in the stdlib. But that a big change... > > For Platform, I fail to see the point of supporting both multiple > use, and comma-separated lists. Right. This is PEP 314 though, so we will need a deprecation cycle. The simplest way to fix this is to use (multiple use) and deprecate comma-separated I guess. > > For Metadata-Version, I think formally, the only legal value according > to the PEP is 1.2. If 1.0 and 1.1 are also conforming values, the PEP > should elaborate what it means to put a different version number into > the field. Right. I'll extend on this Thanks for all this feedback Tarek From cournape at gmail.com Wed Dec 23 13:14:56 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 23 Dec 2009 21:14:56 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912230228q2e75a18ak3f695d1bc8a7785e@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> <5b8d13220912222012p69cdc40arcf262c7c02e0dc7d@mail.gmail.com> <94bdd2610912230228q2e75a18ak3f695d1bc8a7785e@mail.gmail.com> Message-ID: <5b8d13220912230414j112a05f0r642a8815990e7cdd@mail.gmail.com> On Wed, Dec 23, 2009 at 7:28 PM, Tarek Ziad? wrote: > On Wed, Dec 23, 2009 at 5:12 AM, David Cournapeau wrote: >> On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziad? wrote: >> >>> >>> When you say "which could be solved relatively easily" I suggest that >>> you take the time to add concise and precise proposals in >>> bugs.python.org so I can work on them. >> >> Technically, it is easy. Only have two mechanisms for data files: one >> for installed data files, and one for extra source files (as done in >> automake for example): >> ?- ?Extra files only need to be listed (and included in sdist) >> ?- ?Install data files need a target directory. One of the problem >> with distutils here is that you can only hardcode paths - in my own >> packaging solution, I use $path variables so that any path defined >> internally can be reused ($bindir, etc...); something similar could be >> defined in distutils. > > distutils uses install schemes with variables too ($base, etc), that > are expanded at installation time. and differs depending on the > options you pass to the install command. > (look at the command/install module) To be clear: I am talking about the POV of the setup.py writer here. AFAIK, those $path variables are not available in this case: when using data_files, you only have the choice between using absolute files or relative to package path. That's why you had to advise one poster to move his files into a package in one recent email, and that the only solution to another poster was to create a new command (to access those $path vars). The notion of data vs package data does not make much sense IMHO. All current methods should be deprecated, to the profit of the only difference that matters: installed vs non-installed data files. cheers, David From ziade.tarek at gmail.com Wed Dec 23 14:08:08 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 14:08:08 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912230414j112a05f0r642a8815990e7cdd@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> <5b8d13220912222012p69cdc40arcf262c7c02e0dc7d@mail.gmail.com> <94bdd2610912230228q2e75a18ak3f695d1bc8a7785e@mail.gmail.com> <5b8d13220912230414j112a05f0r642a8815990e7cdd@mail.gmail.com> Message-ID: <94bdd2610912230508s26d0035fw6a84a4ea4e4a43b2@mail.gmail.com> On Wed, Dec 23, 2009 at 1:14 PM, David Cournapeau wrote: > On Wed, Dec 23, 2009 at 7:28 PM, Tarek Ziad? wrote: >> On Wed, Dec 23, 2009 at 5:12 AM, David Cournapeau wrote: >>> On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziad? wrote: >>> >>>> >>>> When you say "which could be solved relatively easily" I suggest that >>>> you take the time to add concise and precise proposals in >>>> bugs.python.org so I can work on them. >>> >>> Technically, it is easy. Only have two mechanisms for data files: one >>> for installed data files, and one for extra source files (as done in >>> automake for example): >>> ?- ?Extra files only need to be listed (and included in sdist) >>> ?- ?Install data files need a target directory. One of the problem >>> with distutils here is that you can only hardcode paths - in my own >>> packaging solution, I use $path variables so that any path defined >>> internally can be reused ($bindir, etc...); something similar could be >>> defined in distutils. >> >> distutils uses install schemes with variables too ($base, etc), that >> are expanded at installation time. and differs depending on the >> options you pass to the install command. >> (look at the command/install module) > > To be clear: I am talking about the POV of the setup.py writer here. > AFAIK, those $path variables are not available in this case: when > using data_files, you only have the choice between using absolute > files or relative to package path. That's why you had to advise one > poster to move his files into a package in one recent email, and that > the only solution to another poster was to create a new command (to > access those $path vars). We are discussing these options as a matter of fact, in PEP 376. Maybe you missed the mails related to that. See: http://wiki.python.org/moin/Distutils/DiscussionOverview feel free to comment and help, From martin at v.loewis.de Wed Dec 23 15:09:57 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Dec 2009 15:09:57 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> Message-ID: <4B3224B5.8000503@v.loewis.de> > Right, this is not good because the comma means "and" then "or" > > I see two ways to fix this: > > - introduce groups, and say the the comma is "or" inside a group: > > 0.9, (>1.0, !=1.3.4, <2.0), 3.4 ==> 0.9 or (>1.0 and !=1.3.4 and > <2.0) or 3.4 This I don't like. It may be the disjunctive normal form, but I doubt users will understand it. > - use an explicit separator word (and/or) There is a third solution: don't support "or". The only example in the PEP that assumes "or" (i.e. "2.5, 2.6") could be rewritten as ">=2.5,<2.7". BTW: how is "==" specified? Is "2.5.4" == "2.5"? If not, would "2.5, 2.6" really mean "either Python 2.5 or Python 2.6, but neither 2.5.1 nor 2.6.1"? >> For the Copyright field, I'm not sure what the purpose of it is. >> If it is what I think it is (listing attribution), it should probably >> be specified as "multiple use". > > I just noticed this field was added after PEP 314. I don't think we > should keep it at all. > I don't see a use case for this (README etc.. are enough for copyright matters) +1. > What I have suggested last week was to introduce a delimiter to start > each new line so > this field stays RFC 822 compatible and reST works if we want to read > back the value: > > Description: This module collects votes from beagles > | in order to determine their electoral wishes. > | Do *not* try to use this module with basset hounds; > | it makes them grumpy. > | > | example code : > | > | >>> 2 + 1 > | 3 That may be reasonable, but it's also slightly incompatible. I.e. you'll need to specify what happens if the vertical bar is missing (OTOH, for 1.2, you could actually require it). > A cleaner option of course, is to drop the RFC 822 format and to use a > simpler format like json for example, since we have a reader/writer > for that in the stdlib. That is a weak argument: we have readers and writers for rfc822 in the stdlib also, and that was one of the reasons why this format was originally chosen > But that a big change... I think the RFC822 way would be to use a MIME encoding in the header (either Q or B) to transmit spaces reliable. But I don't think we want that. >> For Platform, I fail to see the point of supporting both multiple >> use, and comma-separated lists. > > Right. This is PEP 314 though, so we will need a deprecation cycle. > The simplest way to fix this is to use (multiple use) and deprecate > comma-separated I guess. Not really. In 1.2, you can specify anything you like. That's the whole point of having a version number in the data. Regards, Martin From ziade.tarek at gmail.com Wed Dec 23 19:45:28 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 19:45:28 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <4B3224B5.8000503@v.loewis.de> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> Message-ID: <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> 2009/12/23 "Martin v. L?wis" : [..] > There is a third solution: don't support "or". The only example in > the PEP that assumes "or" (i.e. "2.5, 2.6") could be rewritten as > ">=2.5,<2.7". Sure, that's even simpler, > > BTW: how is "==" specified? Is "2.5.4" == "2.5"? > If not, would > "2.5, 2.6" really mean "either Python 2.5 or Python 2.6, but neither > 2.5.1 nor 2.6.1"? Yes, 2.5 concerns all 2.5.x versions, but a micro version could be given as well I am changing this [..] >> What I have suggested last week was to introduce a delimiter to start >> each new line so >> this field stays RFC 822 compatible and reST works if we want to read >> back the value: >> >> Description: This module collects votes from beagles >> ? ? ? ? ? ?| in order to determine their electoral wishes. >> ? ? ? ? ? ?| Do *not* try to use this module with basset hounds; >> ? ? ? ? ? ?| it makes them grumpy. >> ? ? ? ? ? ?| >> ? ? ? ? ? ?| example code : >> ? ? ? ? ? ?| >> ? ? ? ? ? ?| ? ? ?>>> 2 + 1 >> ? ? ? ? ? ?| ? ? ?3 > > That may be reasonable, but it's also slightly incompatible. I.e. you'll > need to specify what happens if the vertical bar is missing (OTOH, for > 1.2, you could actually require it). Yes, I was thinking about leaving 1.0 and 1.1 as they are, and add this vertical bar in 1.2. after the 8 spaces chars that are added after the EOLs. >> A cleaner option of course, is to drop the RFC 822 format and to use a >> simpler format like json for example, since we have a reader/writer >> for that in the stdlib. > > That is a weak argument: we have readers and writers for rfc822 in the > stdlib also, and that was one of the reasons why this format was > originally chosen Let me rephrase it: RFC 822 was used because there were no reST fields back then when PEP 314 was introduced. RFC 822 doesn't fit well with what we need today (reST content) because we have to encode the values in something that can be put in a header. Using another encoding that knows how to encode/decode python structure without having to take care of EOLs, space stripping, etc is just more straightforward. And we happen to have a json encoder/decoder in the stdlib. But I am fine with the "|" solution with RFC 822. Regards, Tarek From martin at v.loewis.de Wed Dec 23 19:51:59 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Dec 2009 19:51:59 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> Message-ID: <4B3266CF.7030705@v.loewis.de> >> BTW: how is "==" specified? Is "2.5.4" == "2.5"? >> If not, would >> "2.5, 2.6" really mean "either Python 2.5 or Python 2.6, but neither >> 2.5.1 nor 2.6.1"? > > Yes, 2.5 concerns all 2.5.x versions, but a micro version could be given as well > I am changing this Are you sure this is a good idea? Does that apply to all PEP 386 version numbers? Regards, Martin From ziade.tarek at gmail.com Wed Dec 23 20:23:02 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 20:23:02 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <4B3266CF.7030705@v.loewis.de> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> Message-ID: <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> 2009/12/23 "Martin v. L?wis" : >>> BTW: how is "==" specified? Is "2.5.4" == "2.5"? >>> If not, would >>> "2.5, 2.6" really mean "either Python 2.5 or Python 2.6, but neither >>> 2.5.1 nor 2.6.1"? >> >> Yes, 2.5 concerns all 2.5.x versions, but a micro version could be given as well >> I am changing this > > Are you sure this is a good idea? The use case I see is : an important ssl security version is released in 2.6.5 and a project that uses ssl updates its metadata and releases a new version that makes sure Python 2.5 users have that security patch Requires-Python: >=2.6.5 Then encourages its users to update their versions. > Does that apply to all PEP 386 version numbers? Not sure what you mean here, Regards, Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Wed Dec 23 20:24:34 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 20:24:34 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> Message-ID: <94bdd2610912231124i59e4dab8x83e708ebaa26aaa2@mail.gmail.com> 2009/12/23 Tarek Ziad? : [..] > .. makes sure Python 2.5 users have that security patch s/Python 2.5 users/Python 2.6 users From sridharr at activestate.com Wed Dec 23 20:24:46 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 23 Dec 2009 11:24:46 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> Message-ID: <4B326E7E.3050407@activestate.com> On 12/22/2009 10:15 AM, Lennart Regebro wrote: >> > Another point that I really like about the service is that the distribution >> > pages provide links to many other related services that are run by other >> > volunteers. Take for examplehttp://search.cpan.org/dist/PAR-Repository-Client/ >> > There is a link to cpanforum, specifically the relevant discussion forum for the >> > module at hand. It shows a link to the bug/request tracker with the number of >> > open bugs, the one next to it will display a nice hierarchical (recursive) list >> > of dependencies. > > Right, those third-party services doesn't exist for PyPI. The reason why PyPI does not have such third-party services - I think - is that it lacks the CPAN like simple directory structure that can be easily mirrored using ftp/rsync, to wit: [Steffen Mueller] > My thesis is that the huge success of the CPAN has been facilitated by > two factors[2]. The first is simplicity. When Jarkko Hietaniemi > originally came up with it, the CPAN was (and mostly still is) just an > FTP archive with a by-author directory structure that is mirrored many > times. http://pypi.python.org/simple does not qualify here, for there is no reliable way one can get, say, the source tarball for package 'Foo' and version '0.3'. For an example, see: http://pypi.python.org/simple/Pylons/ The only way to get, say, version 0.9.7's source code is to scrap that page and get the link "0.9.7_home_page" (assuming Pylons-0.9.7.tar.gz is not already available -- which is entire possible for packages in PyPI: missing source tarballs) and further scrap it for download links. Not very friendly for third-party sites to simply update PyPI packages on daily basis (where rsync works very well). Simplicity is nowhere to be found. One solution I can think of is this: make PyPI only do the job of PAUSE as it does for CPAN; and implement a CPAN like simple directory structure to store packages; make PyPI use that as the package data store; deprecate the XmlRpc interface and rely on simple index files - such as http://www.cpan.org/modules/01modules.mtime.html - instead (consequently rethink PEP 381). This is partially implemented for PyPM at ActiveState and I'd be willing to contribute my time towards doing this for PyPI (if at all there is an interest). -srid From jjkunce at gmail.com Wed Dec 23 20:38:58 2009 From: jjkunce at gmail.com (Jeff Kunce) Date: Wed, 23 Dec 2009 11:38:58 -0800 Subject: [Distutils] buildout examples clarification Message-ID: <833f4cd70912231138k742a817crfff741999fd3f9e4@mail.gmail.com> I am using the buildout page on pypi as a reference ? http://pypi.python.org/pypi/zc.buildout What is the python shell used for the examples?? I can follow what is going on, but I'm not familiar with some of the functions used. For example: >>> write(sample_buildout, 'recipes', 'mkdir.py', ...) Sorry if this is a silly question, but I'm stumped. Thanks. From jim at zope.com Wed Dec 23 20:48:42 2009 From: jim at zope.com (Jim Fulton) Date: Wed, 23 Dec 2009 14:48:42 -0500 Subject: [Distutils] buildout examples clarification In-Reply-To: <833f4cd70912231138k742a817crfff741999fd3f9e4@mail.gmail.com> References: <833f4cd70912231138k742a817crfff741999fd3f9e4@mail.gmail.com> Message-ID: <1099b90b0912231148y6db05e9fm88217b747884f41d@mail.gmail.com> On Wed, Dec 23, 2009 at 2:38 PM, Jeff Kunce wrote: > I am using the buildout page on pypi as a reference > ? http://pypi.python.org/pypi/zc.buildout > > What is the python shell used for the examples?? I can follow what is > going on, but I'm not familiar with some of the functions used. For > example: > ?>>> write(sample_buildout, 'recipes', 'mkdir.py', ...) > > Sorry if this is a silly question, but I'm stumped. ?Thanks. np. The way those examples are written is rather awkward to say the least. You might want to check out the buildout tutorial: http://www.buildout.org/docs/tutorial.html In any case, write just writes out a string to a file. The last argument is the string. The other arguments are joined to construct a path. Jim -- Jim Fulton From martin at v.loewis.de Wed Dec 23 21:13:29 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Dec 2009 21:13:29 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <20091219211228.GA10616@laurie.devork> <94bdd2610912191405s46a02da6wcd949bfafa362f77@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> Message-ID: <4B3279E9.4010507@v.loewis.de> Tarek Ziad? wrote: > 2009/12/23 "Martin v. L?wis" : >>>> BTW: how is "==" specified? Is "2.5.4" == "2.5"? >>>> If not, would >>>> "2.5, 2.6" really mean "either Python 2.5 or Python 2.6, but neither >>>> 2.5.1 nor 2.6.1"? >>> Yes, 2.5 concerns all 2.5.x versions, but a micro version could be given as well >>> I am changing this >> Are you sure this is a good idea? > > The use case I see is : an important ssl security version is released > in 2.6.5 and a project > that uses ssl updates its metadata and releases a new version that > makes sure Python 2.5 users have that security patch > > Requires-Python: >=2.6.5 Why is that a use case for "2.5 concerns all 2.5.x versions"? In this example, the version string "2.5" isn't used at all. > Then encourages its users to update their versions. Specifying micro versions is fine with me. I'm worried about the "2.5 concerns all 2.5.x versions" part. >> Does that apply to all PEP 386 version numbers? > > Not sure what you mean here, If I specify Requires-Dist: zope.interface (3.6) and I have zope.interface 3.6.1b4, is the requirement satisfied? What if I have 3.6b4: is it then satisfied? Regards, Martin From martin at v.loewis.de Wed Dec 23 21:18:52 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Dec 2009 21:18:52 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B326E7E.3050407@activestate.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> Message-ID: <4B327B2C.4080605@v.loewis.de> > One solution I can think of is this: make PyPI only do the job of PAUSE > as it does for CPAN; and implement a CPAN like simple directory > structure to store packages; make PyPI use that as the package data > store I don't know what PAUSE is, but I think there is what you want at http://pypi.python.org/packages/ > deprecate the XmlRpc interface and rely on simple index files - > such as http://www.cpan.org/modules/01modules.mtime.html - instead > (consequently rethink PEP 381). This is partially implemented for PyPM > at ActiveState and I'd be willing to contribute my time towards doing > this for PyPI (if at all there is an interest). I don't think I understand what it is that you want, so I don't know whether I'm interested. Regards, Martin From reinout at vanrees.org Wed Dec 23 22:05:15 2009 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 23 Dec 2009 22:05:15 +0100 Subject: [Distutils] buildout examples clarification In-Reply-To: <833f4cd70912231138k742a817crfff741999fd3f9e4@mail.gmail.com> References: <833f4cd70912231138k742a817crfff741999fd3f9e4@mail.gmail.com> Message-ID: On 12/23/09 8:38 PM, Jeff Kunce wrote: > I am using the buildout page on pypi as a reference > http://pypi.python.org/pypi/zc.buildout > > What is the python shell used for the examples? I can follow what is > going on, but I'm not familiar with some of the functions used. For > example: > >>> write(sample_buildout, 'recipes', 'mkdir.py', ...) > > Sorry if this is a silly question, but I'm stumped. Thanks. Those write() and cat() and ls() methods are helper methods that are provided by zc.buildout's own test setup. (Technically: they're injected into the test's globals namespace, look for a ``test.globs['write'] = some_write_method`` in test*.py if you want the details) Really handy when testing zc.buildout. But they might just be a "little bit hidden" away in the test setup code ;-) Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets" From ziade.tarek at gmail.com Wed Dec 23 22:11:31 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 22:11:31 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <4B3279E9.4010507@v.loewis.de> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> <4B3279E9.4010507@v.loewis.de> Message-ID: <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> 2009/12/23 "Martin v. L?wis" : > Why is that a use case for "2.5 concerns all 2.5.x versions"? > In this example, the version string "2.5" isn't used at all. You asked "Are you sure this is a good idea?" when I said "micro version could be given as well", so I gave you an example where a micro version is required. [..] > > Specifying micro versions is fine with me. I'm worried about > the "2.5 concerns all 2.5.x versions" part. Some applications doesn't care about the micro version, but will care about the minor version because those can introduce API and syntax changes. e.g. If at some point I have a Python 2.6 program that uses the "with" keyword, and I don't want to introduce a __future__ import in my code, I'll just use : Requires-Python: 2.6 And I will not care much about the micro version in that case, since Python will not change its syntax/API in a micro release. >>> Does that apply to all PEP 386 version numbers? >> >> Not sure what you mean here, > > If I specify > > Requires-Dist: zope.interface (3.6) > > and I have zope.interface 3.6.1b4, is the requirement satisfied? > What if I have 3.6b4: is it then satisfied? 3.6 would include all 3.6.x releases as well. So 3.6b4 is excluded since it does not belong to the 3.6 series, but 3.6.1b4 is included. IOW we compare the distribution version only with what is provided in the metadata field and truncate the rest. So here MAJOR.MINOR[pre-release tag] That doesn't concerns the Version field of course. Regards Tarek From jim at zope.com Wed Dec 23 22:14:06 2009 From: jim at zope.com (Jim Fulton) Date: Wed, 23 Dec 2009 16:14:06 -0500 Subject: [Distutils] buildout examples clarification In-Reply-To: References: <833f4cd70912231138k742a817crfff741999fd3f9e4@mail.gmail.com> Message-ID: <1099b90b0912231314q79b38917p7ad665d75f3c1f16@mail.gmail.com> On Wed, Dec 23, 2009 at 4:05 PM, Reinout van Rees wrote: > On 12/23/09 8:38 PM, Jeff Kunce wrote: >> >> I am using the buildout page on pypi as a reference >> ? http://pypi.python.org/pypi/zc.buildout >> >> What is the python shell used for the examples? ?I can follow what is >> going on, but I'm not familiar with some of the functions used. For >> example: >> ? >>> ?write(sample_buildout, 'recipes', 'mkdir.py', ...) >> >> Sorry if this is a silly question, but I'm stumped. ?Thanks. > > Those write() and cat() and ls() methods are helper methods that are > provided by zc.buildout's own test setup. > > > (Technically: they're injected into the test's globals namespace, look for a > ``test.globs['write'] = some_write_method`` in test*.py if you want the > details) > > Really handy when testing zc.buildout. But they might just be a "little bit > hidden" away in the test setup code ;-) The buildout tests (and future real documentation) would benefit greatly from manuel, which would allow me to get rid of all (or most) of these helper functions and make the examples far more readable. So much to do ... Jim -- Jim Fulton From ziade.tarek at gmail.com Wed Dec 23 22:17:41 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 22:17:41 +0100 Subject: [Distutils] buildout examples clarification In-Reply-To: References: <833f4cd70912231138k742a817crfff741999fd3f9e4@mail.gmail.com> Message-ID: <94bdd2610912231317g2596027er36604dcef13314c8@mail.gmail.com> On Wed, Dec 23, 2009 at 10:05 PM, Reinout van Rees wrote: > On 12/23/09 8:38 PM, Jeff Kunce wrote: >> >> I am using the buildout page on pypi as a reference >> ? http://pypi.python.org/pypi/zc.buildout >> >> What is the python shell used for the examples? ?I can follow what is >> going on, but I'm not familiar with some of the functions used. For >> example: >> ? >>> ?write(sample_buildout, 'recipes', 'mkdir.py', ...) >> >> Sorry if this is a silly question, but I'm stumped. ?Thanks. > > Those write() and cat() and ls() methods are helper methods that are > provided by zc.buildout's own test setup. > > > (Technically: they're injected into the test's globals namespace, look for a > ``test.globs['write'] = some_write_method`` in test*.py if you want the > details) > > Really handy when testing zc.buildout. But they might just be a "little bit > hidden" away in the test setup code ;-) If it's not the case already, a small section on these APIs at buildout.org, and a link to it in zc.buildout doc could help understand it imho Regards Tarek From sridharr at activestate.com Wed Dec 23 22:19:07 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 23 Dec 2009 13:19:07 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B327B2C.4080605@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> Message-ID: <4B32894B.8090109@activestate.com> On 12/23/2009 12:18 PM, "Martin v. L?wis" wrote: >> One solution I can think of is this: make PyPI only do the job of PAUSE >> > as it does for CPAN; and implement a CPAN like simple directory >> > structure to store packages; make PyPI use that as the package data >> > store > I don't know what PAUSE is, but I think there is what you want at PAUSE is the web management interface that allows one to upload/manage packages in CPAN. http://pause.perl.org/ -- similar to PyPI (minus the storage part). > > http://pypi.python.org/packages/ > >> > deprecate the XmlRpc interface and rely on simple index files - >> > such ashttp://www.cpan.org/modules/01modules.mtime.html - instead >> > (consequently rethink PEP 381). This is partially implemented for PyPM >> > at ActiveState and I'd be willing to contribute my time towards doing >> > this for PyPI (if at all there is an interest). > I don't think I understand what it is that you want, so I don't know > whether I'm interested. What /packages/source/ lacks is: 1/ Missing packages (eg: Twisted is not there); which is why easy_install/pip had to resolve to scrapping project webpages for guessing download links. In CPAN, almost all module authors upload their sources via PAUSE. 2/ No metadata: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for latest version, b) get the source for a particular version? In CPAN [cpan.org/modules/by-module/AppConfig/ABW/], each tarball has a .meta file describing the module metadata (similar to PKG-INFO). I don't want XmlRpc, but just files/directories (note simplicity in Steffen's post). The former is more of a community issue. Often Python package authors are not using `sdist upload` (whereas this seems to be the convention in the Perl world). The later is what is most relevant to PyPM (or any thirdparty service/tool). 1 and 2 combined makes it possible for anyone willing to write a third-party PyPI functionality to simply rsync the entire PyPI store and begin implementing the desired features like *.cpan.org (eg: test.pypi.org, quality.pypi.org, etc..) What this means is that PyPI has to serve the purpose of being a central package repository (like CPAN) by a) disallowing mere listings (without sources) and requiring sources to be stored in the server, b) storing the metadata along with the sources (so anyone processing it wouldn't have to extract the source and rely on a PKG-INFO file - which may or may not exist). Tools would consequently use the new /packages/sources store (with full metadata and all registered package sources) without having to resolve to webpage scrapping hacks. -srid PS: Our internal mirror is similar to /packages/sources except it (a) also contains external packages (eg: Twisted), (b) and pre-extracts PKG-INFO and requires.txt out of the source tarballs. Everyday, it uses PyPI's XmlRpc interface to re-download (using easy_install scrapping logic) the recently releases packages From sridharr at activestate.com Wed Dec 23 22:30:53 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 23 Dec 2009 13:30:53 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B32894B.8090109@activestate.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> Message-ID: <4B328C0D.7080705@activestate.com> On 12/23/2009 1:19 PM, Sridhar Ratnakumar wrote: > > What /packages/source/ lacks is: > > 1/ Missing packages (eg: Twisted is not there); which is why > easy_install/pip had to resolve to scrapping project webpages for > guessing download links. In CPAN, almost all module authors upload their > sources via PAUSE. > > 2/ No metadata: When only source tarballs are stored > [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to > a) get the source for latest version, b) get the source for a particular > version? In CPAN [cpan.org/modules/by-module/AppConfig/ABW/], each > tarball has a .meta file describing the module metadata (similar to > PKG-INFO). I don't want XmlRpc, but just files/directories (note > simplicity in Steffen's post). 3/ Indices such as http://www.cpan.org/modules/01modules.mtime.html (or TXT files) to get a) recently released packages, b) list of release versions for a particular package, and so on (which would obviate the XmlRpc interface) -srid From regebro at gmail.com Wed Dec 23 22:33:49 2009 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 23 Dec 2009 22:33:49 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B326E7E.3050407@activestate.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> Message-ID: <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> On Wed, Dec 23, 2009 at 20:24, Sridhar Ratnakumar wrote: > The reason why PyPI does not have such third-party services - I think - is > that it lacks the CPAN like simple directory structure that can be easily > mirrored using ftp/rsync, to wit: Nah, you can do that via /packages/, there is also an API to get the metadata for a package. I think in general it's not an API problem. I think it's partly a problem that nobody has thunk the thought. I think the idea of a site with automatically generated documentation for *every* package is interesting. But I don't have time to work on that right now. Talk to me again in six months, then I might have time for another free-time project. :) > 1/ Missing packages (eg: Twisted is not there) The Twisted guys do not upload their packages to PyPI. I think that's a mistake, but it's hardly PyPI's fault. There is no law saying you have to use CPAN either. > 2/ No metadata: When only source tarballs are stored > [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) > get the source for latest version Download it from the above location. > b) get the source for a particular Download it from the above location. > version? In CPAN [cpan.org/modules/by-module/AppConfig/ABW/], each tarball > has a .meta file describing the module metadata (similar to PKG-INFO). http://pypi.python.org/pypi?:action=doap&name=Twisted%20Mail&version=9.0.0 This is not a problem about missing API or functionality, but that you don't know about it. In the last case that link exists at the bottom of every package page. And you see how it works. > don't want XmlRpc, but just files/directories (note simplicity in Steffen's > post). It's not XML-RPC because the metadata file is in XML-format. But yes, you can't duplicate both the files and the metadata in one go, you have to do it separately. But that then begs the question: How often do you need to do both? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cournape at gmail.com Wed Dec 23 23:12:54 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 24 Dec 2009 07:12:54 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912230508s26d0035fw6a84a4ea4e4a43b2@mail.gmail.com> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> <5b8d13220912222012p69cdc40arcf262c7c02e0dc7d@mail.gmail.com> <94bdd2610912230228q2e75a18ak3f695d1bc8a7785e@mail.gmail.com> <5b8d13220912230414j112a05f0r642a8815990e7cdd@mail.gmail.com> <94bdd2610912230508s26d0035fw6a84a4ea4e4a43b2@mail.gmail.com> Message-ID: <5b8d13220912231412y41abf6abt1c85a3d196a7f206@mail.gmail.com> On Wed, Dec 23, 2009 at 10:08 PM, Tarek Ziad? wrote: > On Wed, Dec 23, 2009 at 1:14 PM, David Cournapeau wrote: >> On Wed, Dec 23, 2009 at 7:28 PM, Tarek Ziad? wrote: >>> On Wed, Dec 23, 2009 at 5:12 AM, David Cournapeau wrote: >>>> On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziad? wrote: >>>> >>>>> >>>>> When you say "which could be solved relatively easily" I suggest that >>>>> you take the time to add concise and precise proposals in >>>>> bugs.python.org so I can work on them. >>>> >>>> Technically, it is easy. Only have two mechanisms for data files: one >>>> for installed data files, and one for extra source files (as done in >>>> automake for example): >>>> ?- ?Extra files only need to be listed (and included in sdist) >>>> ?- ?Install data files need a target directory. One of the problem >>>> with distutils here is that you can only hardcode paths - in my own >>>> packaging solution, I use $path variables so that any path defined >>>> internally can be reused ($bindir, etc...); something similar could be >>>> defined in distutils. >>> >>> distutils uses install schemes with variables too ($base, etc), that >>> are expanded at installation time. and differs depending on the >>> options you pass to the install command. >>> (look at the command/install module) >> >> To be clear: I am talking about the POV of the setup.py writer here. >> AFAIK, those $path variables are not available in this case: when >> using data_files, you only have the choice between using absolute >> files or relative to package path. That's why you had to advise one >> poster to move his files into a package in one recent email, and that >> the only solution to another poster was to create a new command (to >> access those $path vars). > > We are discussing these options as a matter of fact, in PEP 376. I don't see data files mentioned in PEP 376, nor how the PEP is related to this discussion. David From ziade.tarek at gmail.com Wed Dec 23 23:18:36 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 23:18:36 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912231412y41abf6abt1c85a3d196a7f206@mail.gmail.com> References: <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> <5b8d13220912222012p69cdc40arcf262c7c02e0dc7d@mail.gmail.com> <94bdd2610912230228q2e75a18ak3f695d1bc8a7785e@mail.gmail.com> <5b8d13220912230414j112a05f0r642a8815990e7cdd@mail.gmail.com> <94bdd2610912230508s26d0035fw6a84a4ea4e4a43b2@mail.gmail.com> <5b8d13220912231412y41abf6abt1c85a3d196a7f206@mail.gmail.com> Message-ID: <94bdd2610912231418o70ddb7f1ge3a3ad83817227dd@mail.gmail.com> On Wed, Dec 23, 2009 at 11:12 PM, David Cournapeau wrote: [..] >> We are discussing these options as a matter of fact, in PEP 376. > > I don't see data files mentioned in PEP 376, nor how the PEP is > related to this discussion. The PEP contains what has been already discussed. As said earlier, take a look at http://wiki.python.org/moin/Distutils/DiscussionOverview for what is being currently discussed. From glyph at twistedmatrix.com Wed Dec 23 23:20:44 2009 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 23 Dec 2009 17:20:44 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> Message-ID: On Dec 23, 2009, at 4:33 PM, Lennart Regebro wrote: >> >> 1/ Missing packages (eg: Twisted is not there) > > The Twisted guys do not upload their packages to PyPI. I think that's > a mistake, but it's hardly PyPI's fault. There is no law saying you > have to use CPAN either. For what it's worth, we don't upload because it's a big pain, and nobody cares anyway. It's a big pain because there are two ways to upload, and neither one works for us. We can't use 'setup.py upload' because we don't use 'sdist' to produce our tarball releases (although a discussion of why 'sdist' is insufficient is a topic for another post). The other way to upload, manually interacting with a form in a web browser, is annoying and as far as I know it is hostile to automation. Nobody cares because you can 'easy_install twisted' already, and you can find Twisted on the PyPI web page. It's not clear to us what benefits uploading to PyPI would have beyond that. If someone would like to give us a good reason to upload or explain how uploading might be made easy, maybe we'd start doing it :). -------------- next part -------------- An HTML attachment was scrubbed... URL: From kiorky at cryptelium.net Wed Dec 23 23:25:27 2009 From: kiorky at cryptelium.net (kiorky) Date: Wed, 23 Dec 2009 23:25:27 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> Message-ID: <4B3298D7.6040706@cryptelium.net> Lennart Regebro a ?crit : > On Wed, Dec 23, 2009 at 20:24, Sridhar Ratnakumar > wrote: > > The Twisted guys do not upload their packages to PyPI. I think that's The tiny 10MB upload is a blocker i think also. For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From sridharr at activestate.com Wed Dec 23 23:28:10 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 23 Dec 2009 14:28:10 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> Message-ID: <4B32997A.3070407@activestate.com> On 12/23/2009 1:33 PM, Lennart Regebro wrote: > On Wed, Dec 23, 2009 at 20:24, Sridhar Ratnakumar > wrote: >> The reason why PyPI does not have such third-party services - I think - is >> that it lacks the CPAN like simple directory structure that can be easily >> mirrored using ftp/rsync, to wit: > > Nah, you can do that via /packages/, there is also an API to get the > metadata for a package. I think in general it's not an API problem. It is indeed technically possible to do that with the PyPI XmlRpc API alone; but what I was referring to is enabling the mindset: a simple *self-contained* (i.e., without having to use an API to get metadata) directory structure that can simply be mirrored by using existing tools like rsync could *enable* developers interested in providing extending packaging functionality such as testing, quality measurements, documents, search, etc... to easily create such sites and maintain it. At least, this is what - I understand - happened in the Perl community. > I think it's partly a problem that nobody has thunk the thought. I > think the idea of a site with automatically generated documentation > for *every* package is interesting. But I don't have time to work on > that right now. Talk to me again in six months, then I might have time > for another free-time project. :) > >> 1/ Missing packages (eg: Twisted is not there) > > The Twisted guys do not upload their packages to PyPI. I think that's > a mistake, but it's hardly PyPI's fault. There is no law saying you > have to use CPAN either. Yes, as I said it is more of a community issue (than a PyPI one). What I also did mention was that because of this community issue, tools like easy_install/pip had to resolve to scrapping project webpages for guessing download links in an adhoc fashion. Also, further below the mail, I suggested PyPI to disallow mere project listings (without sources) and require sources to be stored in the server. One way to achieve this is requiring package authors to use the `sdist upload` toolchain which automatically creates a source tarball including metadata (in case one forgets to include it). >> 2/ No metadata: When only source tarballs are stored >> [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) >> get the source for latest version > > Download it from the above location. > >> b) get the source for a particular > > Download it from the above location. Perhaps if I were to rephrase the question, it would be clear this time: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for the latest version (when the /P/Pylons contains multiple versions -- in other words, how do I find the later version in first place?), b) get the source for a particular version (without having to construct the filename, or do a adhoc matching with filenames to guess that Pylons-1.2.3.tar.gz corresponds to version 1.2.3)? If the answer is to do a HTTP GET first, then please see the next response. >> version? In CPAN [cpan.org/modules/by-module/AppConfig/ABW/], each tarball >> has a .meta file describing the module metadata (similar to PKG-INFO). > > http://pypi.python.org/pypi?:action=doap&name=Twisted%20Mail&version=9.0.0 > > This is not a problem about missing API or functionality, but that you > don't know about it. In the last case that link exists at the bottom > of every package page. And you see how it works. As the CPAN .meta example was given in the context of having a simple directory structure that can be mirrored using existing tools like rsync, what I was pointing out is the lack of such an implementation, not the functionality itself (which, as you have shown, is currently supported by doing a HTTP GET that would return a XML content -- not something that is rsync-friendly). >> don't want XmlRpc, but just files/directories (note simplicity in Steffen's >> post). > > It's not XML-RPC because the metadata file is in XML-format. While the specific case mentioned above (metadata for a specific or the latest version of a package) uses HTTP GET and XML, generally speaking .. to get a) the list of recently releases, b) list of all versions of a package, one has to use the XmlRpc API methods `changelog` and `package_releases` respectively. > But yes, you can't duplicate both the files and the metadata in one > go, you have to do it separately. But that then begs the question: How > often do you need to do both? As often as the mirror sites would update their content (i.e., one or more times a day). As often as the (future) third-party sites update their PyPI content (source + metadata). One such user is the PyPM backend itself which at the moment uses the XmlRpc to pull data from PyPI on a daily basis. -srid From ziade.tarek at gmail.com Wed Dec 23 23:32:42 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 23 Dec 2009 23:32:42 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> Message-ID: <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> On Wed, Dec 23, 2009 at 11:20 PM, Glyph Lefkowitz wrote: > > On Dec 23, 2009, at 4:33 PM, Lennart Regebro wrote: > > 1/ Missing packages (eg: Twisted is not there) > > The Twisted guys do not upload their packages to PyPI. I think that's > a mistake, but it's hardly PyPI's fault. There is no law saying you > have to use CPAN either. > > For what it's worth, we don't upload because it's a big pain, and nobody > cares anyway. > It's a big pain because there are two ways to upload, and neither one works > for us. ?We can't use 'setup.py upload' because we don't use 'sdist' to > produce our tarball releases (although a discussion of why 'sdist' is > insufficient is a topic for another post). ?The other way to upload, > manually interacting with a form in a web browser, is annoying and as far as > I know it is hostile to automation. Note that it wouldn't take long to override the upload command so it works independantly from any other *dist command. I could even add a --dist-file option to it so you can point an existing archive to push at PyPI, so running sdist or another *dist command wouldn't be mandatory anymore From sridharr at activestate.com Wed Dec 23 23:51:18 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 23 Dec 2009 14:51:18 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> Message-ID: <4B329EE6.1000801@activestate.com> On 12/23/2009 2:32 PM, Tarek Ziad? wrote: > On Wed, Dec 23, 2009 at 11:20 PM, Glyph Lefkowitz > wrote: >> > >> > On Dec 23, 2009, at 4:33 PM, Lennart Regebro wrote: >> > >> > 1/ Missing packages (eg: Twisted is not there) >> > >> > The Twisted guys do not upload their packages to PyPI. I think that's >> > a mistake, but it's hardly PyPI's fault. There is no law saying you >> > have to use CPAN either. >> > >> > For what it's worth, we don't upload because it's a big pain, and nobody >> > cares anyway. >> > It's a big pain because there are two ways to upload, and neither one works >> > for us. We can't use 'setup.py upload' because we don't use 'sdist' to >> > produce our tarball releases (although a discussion of why 'sdist' is >> > insufficient is a topic for another post). The other way to upload, >> > manually interacting with a form in a web browser, is annoying and as far as >> > I know it is hostile to automation. > Note that it wouldn't take long to override the upload command so it works > independantly from any other *dist command. > > I could even add a --dist-file option to it so you can point an > existing archive to push at PyPI, > so running sdist or another *dist command wouldn't be mandatory anymore The advantage of sdist is that the metadata (PKG-INFO; .egg-info/*.txt) is automatically included in the source distribution. For a manually generated source distribution, this is not always the case. Consequently, one is forced to run the `egg_info` command ..something that is unacceptable if you do not want to run Python code in a server that simply mirrors PyPI. It would be much interesting to hear arguments for why `sdist` is not suited for Twisted releases and see if it can be fixed. -srid From ziade.tarek at gmail.com Thu Dec 24 00:13:32 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 00:13:32 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B329EE6.1000801@activestate.com> References: <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <4B329EE6.1000801@activestate.com> Message-ID: <94bdd2610912231513t49936beekd7c9bd72fe8f50fb@mail.gmail.com> On Wed, Dec 23, 2009 at 11:51 PM, Sridhar Ratnakumar wrote: [..] >> Note that it wouldn't take long to override the upload command so it works >> independantly from any other *dist command. >> >> I could even add a --dist-file option to it so you can point an >> existing archive to push at PyPI, >> so running sdist or another *dist command wouldn't be mandatory anymore > > The advantage of sdist is that the metadata (PKG-INFO; .egg-info/*.txt) is > automatically included in the source distribution. For a manually generated > source distribution, this is not always the case. Consequently, one is > forced to run the `egg_info` command ..something that is unacceptable if you > do not want to run Python code in a server that simply mirrors PyPI. For the Twisted case, we are talking about the very same archive that is downloaded by the installers from their websites, so its just a matter of uploaded an existing archive. What you are referring to is more likely to be QA controls, and running sdist doesn't really prevent from uploading low-QA packages. The new check command in Distutils (that is run before register or sdist is called for example) is the place where we can add QA controls > > It would be much interesting to hear arguments for why `sdist` is not suited > for Twisted releases and see if it can be fixed. It's orthogonal to having a standalone upload command, but I agree it's interesting to know. -- Tarek Ziad? | http://ziade.org From david.lyon at preisshare.net Thu Dec 24 00:10:29 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 23 Dec 2009 18:10:29 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912230108q274ab3cbr9015d2ac4cbbd121@mail.gmail.com> References: <5b8d13220912211448y3e309380je72d67a368ec276e@mail.gmail.com> <94bdd2610912211604o27bd1656q9915ae533895fc29@mail.gmail.com> <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> <319e029f0912221601o4dac1df4kff1414fef26ec3dc@mail.gmail.com> <51c331ec3db84cceb8f2a8cf911a9aac@preisshare.net> <319e029f0912230108q274ab3cbr9015d2ac4cbbd121@mail.gmail.com> Message-ID: On Wed, 23 Dec 2009 10:08:24 +0100, Lennart Regebro wrote: >> There is no .tar.gz, .zip, .bz2, .exe, .msi or .egg concept of >> packages in perl. And having to pick one.. that may or may not >> be right for your configuration. > > So your usecase, that a Windows user refuses to install anything else > than .msi files is solved in CPAN by the user not installing using > CPAN. Well, that's handy. And the Python situation is worse how, > exactly? The same operations take a longer time to complete than the equivalent operations do with perl. > That is the most ridicolous excuse I've heard in a long time. People > are saying to Guido that "Python needs CPAN" and wonders what that > means. You come with you usual anti distutils rant, I ask *you* what > the connection is, and you tell med to go hassle Guido. Yeah, I did. or maybe you can google on it.. > You are not the only one who says it needs to be rewritten from > scratch. If you are saying that the few people that exists that agree > with you about this *also* is not enough, then you are saying that the > community does not agree with you that it needs to be rewritten. I have no strong views on whether distutils needs to be rewritten now or not. If anything, I'm probably of the opinion that most of the bugs and problems could be corrected with time. >> And if python management aren't united on the need for it, >> doing a distutils replacement becomes an even weaker >> proposition. > > So you are saying that you think distutils must be replaced, and that > the only persons who understand why it needs to be replaced are not > willing or able to do or even start the work. I said that distutils could do with a revamp. >> PEP-345 has been open since 2005. In CPAN it just would >> have been a beer fight and have been over in a weekend. > > Maybe they have less people who complain, and use up their and other > peoples energies by ranting and complaining instead of coding. They do. You are right. > In any case, I take your answer as a definite statement from your side > that you will not help neither fix nor replace distutils. And then I > honestly see no reason why I should listen to you anymore. Well that would be an erroneous statement. I recently made a PEP proposal recently about static metadata. I also try to make submissions to the python tracker when I can. I also write code. I would love to see a CPAN style framework in place for python but I'm not under any dilusions about the enormity of the task. One of the great challenges is that most of the discussion is 'bottom-up', rather than being 'top-down'. It's very difficult to rebuild the python packaging framework bottom up. That is, by doing it with patches on the python tracker, and incremental PEPs. Without having some sort of master plan. And your rant about 'just go do it' falls on my deaf ears. If this CPAN thing is so important to python then it stands to reason that people would get invited to a meeting about to at least discuss it it. So if nobody in python management can afford a case of beer to get people together to discuss it all in person then how can volunteers on a mailing list be expected to take it all too seriously. David From david.lyon at preisshare.net Thu Dec 24 01:55:20 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 23 Dec 2009 19:55:20 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B32997A.3070407@activestate.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B32997A.3070407@activestate.com> Message-ID: <29159c62e88822c93a805cd60d993336@preisshare.net> Hi Srid, On Wed, 23 Dec 2009 14:28:10 -0800, Sridhar Ratnakumar wrote: > .. what I was referring to is enabling the mindset: a simple > *self-contained* (i.e., without having to use an API to get metadata) > directory structure that can simply be mirrored by using existing tools > like rsync could *enable* developers interested in providing extending > packaging functionality such as testing, quality measurements, > documents, search, etc... to easily create such sites and maintain it. .. >> I think it's partly a problem that nobody has thunk the thought. They have. I'd say a lot of work has been done looking at the code. See: http://www.pycheesecake.org/ I've been working on running and extending that package testing. My tool is at: http://bitbucket.org/djlyon/pypi-package-testbot/ It's a very time consuming job to test all the packages. Taking at least a few hours a day. As I understand it pythoners got a little offended by the language that was used in it (liberal english - and importing of perl terms) Maybe the use of 'cheesy' fell out of fashion. I don't know. I've downloaded the package and run it against a few packages and it seems to be worthwhile. Some of the terms used in the output reports are pretty easy to change. >> think the idea of a site with automatically generated documentation >> for *every* package is interesting. We need to be looking at stuff like this sure. David From exarkun at twistedmatrix.com Thu Dec 24 03:59:28 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 24 Dec 2009 02:59:28 -0000 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> Message-ID: <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> On 23 Dec, 10:32 pm, ziade.tarek at gmail.com wrote: >On Wed, Dec 23, 2009 at 11:20 PM, Glyph Lefkowitz > wrote: >> >>On Dec 23, 2009, at 4:33 PM, Lennart Regebro wrote: >> >>1/ Missing packages (eg: Twisted is not there) >> >>The Twisted guys do not upload their packages to PyPI. I think that's >>a mistake, but it's hardly PyPI's fault. There is no law saying you >>have to use CPAN either. >> >>For what it's worth, we don't upload because it's a big pain, and >>nobody >>cares anyway. >>It's a big pain because there are two ways to upload, and neither one >>works >>for us. ?We can't use 'setup.py upload' because we don't use 'sdist' >>to >>produce our tarball releases (although a discussion of why 'sdist' is >>insufficient is a topic for another post). ?The other way to upload, >>manually interacting with a form in a web browser, is annoying and as >>far as >>I know it is hostile to automation. > >Note that it wouldn't take long to override the upload command so it >works >independantly from any other *dist command. Let's leave aside how long it would take to override the command (ie, implement a new command-line tool to upload a package to PyPI). There have been a few responses to Glyph's mention that "setup.py upload" doesn't work for Twisted. I'm much more curious to hear replies to his other point - "nobody cares anyway". What's with the interest in having packages hosted on PyPI? I'm not specifically opposed to this, but I don't see any reason it would benefit anyone either. It'd be awesome if someone could explain this. Perhaps if the answer were included somewhere on PyPI or in the distutils documentation, more people would see the light and upload their packages. Jean-Paul From jjkunce at gmail.com Thu Dec 24 04:06:43 2009 From: jjkunce at gmail.com (Jeff Kunce) Date: Wed, 23 Dec 2009 19:06:43 -0800 Subject: [Distutils] buildout examples clarification In-Reply-To: <94bdd2610912231317g2596027er36604dcef13314c8@mail.gmail.com> References: <833f4cd70912231138k742a817crfff741999fd3f9e4@mail.gmail.com> <94bdd2610912231317g2596027er36604dcef13314c8@mail.gmail.com> Message-ID: <833f4cd70912231906v77cfa7b8j9dff9a71f6f2cfae@mail.gmail.com> Thanks for all the responses. I could understand the examples - and wanted to get at the tools that were used. Your pointers are helpful. I was trying to understand some finer points of buildout, without necessarily learning all its internal workings. The tests, of course, would be a good place for me to start. -- Jeff On Wed, Dec 23, 2009 at 1:17 PM, Tarek Ziad? wrote: > On Wed, Dec 23, 2009 at 10:05 PM, Reinout van Rees wrote: >> On 12/23/09 8:38 PM, Jeff Kunce wrote: >>> >>> I am using the buildout page on pypi as a reference >>> ? http://pypi.python.org/pypi/zc.buildout >>> >>> What is the python shell used for the examples? ?I can follow what is >>> going on, but I'm not familiar with some of the functions used. For >>> example: >>> ? >>> ?write(sample_buildout, 'recipes', 'mkdir.py', ...) >>> >>> Sorry if this is a silly question, but I'm stumped. ?Thanks. >> >> Those write() and cat() and ls() methods are helper methods that are >> provided by zc.buildout's own test setup. >> >> >> (Technically: they're injected into the test's globals namespace, look for a >> ``test.globs['write'] = some_write_method`` in test*.py if you want the >> details) >> >> Really handy when testing zc.buildout. But they might just be a "little bit >> hidden" away in the test setup code ;-) > > If it's not the case already, a small section on these APIs at > buildout.org, and a link to it in zc.buildout doc could help > understand it imho > > Regards > Tarek > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From exarkun at twistedmatrix.com Thu Dec 24 04:11:55 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 24 Dec 2009 03:11:55 -0000 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B329EE6.1000801@activestate.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <4B329EE6.1000801@activestate.com> Message-ID: <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> On 23 Dec, 10:51 pm, sridharr at activestate.com wrote: >On 12/23/2009 2:32 PM, Tarek Ziad? wrote: >>On Wed, Dec 23, 2009 at 11:20 PM, Glyph Lefkowitz >> wrote: >>> > >>> > On Dec 23, 2009, at 4:33 PM, Lennart Regebro wrote: >>> > >>> > 1/ Missing packages (eg: Twisted is not there) >>> > >>> > The Twisted guys do not upload their packages to PyPI. I think >>>that's >>> > a mistake, but it's hardly PyPI's fault. There is no law saying >>>you >>> > have to use CPAN either. >>> > >>> > For what it's worth, we don't upload because it's a big pain, and >>>nobody >>> > cares anyway. >>> > It's a big pain because there are two ways to upload, and neither >>>one works >>> > for us. We can't use 'setup.py upload' because we don't use >>>'sdist' to >>> > produce our tarball releases (although a discussion of why 'sdist' >>>is >>> > insufficient is a topic for another post). The other way to >>>upload, >>> > manually interacting with a form in a web browser, is annoying and >>>as far as >>> > I know it is hostile to automation. >>Note that it wouldn't take long to override the upload command so it >>works >>independantly from any other *dist command. >> >>I could even add a --dist-file option to it so you can point an >>existing archive to push at PyPI, >>so running sdist or another *dist command wouldn't be mandatory >>anymore > >The advantage of sdist is that the metadata (PKG-INFO; .egg-info/*.txt) >is automatically included in the source distribution. For a manually >generated source distribution, this is not always the case. >Consequently, one is forced to run the `egg_info` command ..something >that is unacceptable if you do not want to run Python code in a server >that simply mirrors PyPI. > >It would be much interesting to hear arguments for why `sdist` is not >suited for Twisted releases and see if it can be fixed. Release packages for Twisted are constructed using some extra file- finding logic that sdist doesn't provide. Additionally, for years distutils was seen as a blind alley, so we didn't bother to try to create a distutils-friendly substitute. Partially because it seems that distutils has turned a corner over the last year, we are actually (slowly, with difficulty) working towards a more distutils-integrated solution (we're going to try to override sdist with a command based on our existing custom file-finding code). This may result in something we can use with "setup.py upload" (but this isn't currently the primary goal, it would just be a happy side effect). A separate issue with "setup.py upload", though, is that it really wants one of two undesirable things: * the upload is done at the same time as the release package is generated * the release package is generated twice The former requires that proper credentials are available to whoever is creating the release package. Historically for Twisted, this isn't how things have been set up. We could probably deal with it, but it would be nice if it were not a requirement. The latter, of course, introduces the possibility of skew between two release packages, doubling testing requirements, etc. Jean-Paul From cournape at gmail.com Thu Dec 24 04:32:06 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 24 Dec 2009 12:32:06 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912231418o70ddb7f1ge3a3ad83817227dd@mail.gmail.com> References: <319e029f0912212343u22dca6e4v72b21470eb011f5e@mail.gmail.com> <5b8d13220912220502l4daaf9b2t34625d0a2d49023f@mail.gmail.com> <94bdd2610912220535n5289c89bhc24d6fa8effa0497@mail.gmail.com> <5b8d13220912222012p69cdc40arcf262c7c02e0dc7d@mail.gmail.com> <94bdd2610912230228q2e75a18ak3f695d1bc8a7785e@mail.gmail.com> <5b8d13220912230414j112a05f0r642a8815990e7cdd@mail.gmail.com> <94bdd2610912230508s26d0035fw6a84a4ea4e4a43b2@mail.gmail.com> <5b8d13220912231412y41abf6abt1c85a3d196a7f206@mail.gmail.com> <94bdd2610912231418o70ddb7f1ge3a3ad83817227dd@mail.gmail.com> Message-ID: <5b8d13220912231932i979f9e1p72738a68f866b6d2@mail.gmail.com> On Thu, Dec 24, 2009 at 7:18 AM, Tarek Ziad? wrote: > On Wed, Dec 23, 2009 at 11:12 PM, David Cournapeau wrote: > [..] >>> We are discussing these options as a matter of fact, in PEP 376. >> >> I don't see data files mentioned in PEP 376, nor how the PEP is >> related to this discussion. > > The PEP contains what has been already discussed. ?As said earlier, > take a look at http://wiki.python.org/moin/Distutils/DiscussionOverview > for what is being currently discussed. Since I don't see data files mentioned anywhere,s hould I understand the answer as putting a new section for data files in the wiki, or would you prefer that I start a new thread here ? David From ssteinerx at gmail.com Thu Dec 24 06:06:03 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 24 Dec 2009 00:06:03 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> Message-ID: <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> On Dec 23, 2009, at 9:59 PM, exarkun at twistedmatrix.com wrote: > What's with the interest in having packages hosted on PyPI? > > I'm not specifically opposed to this, but I don't see any reason it would benefit anyone either. It'd be awesome if someone could explain this. Perhaps if the answer were included somewhere on PyPI or in the distutils documentation, more people would see the light and upload their packages. I really don't think it has to do with 'having packages hosted on PyPI' as it is in having a single way to install any Python 'thing.' Right now, installing e.g. Twisted, requires finding the website, figuring out which exact file to download, then figuring out exactly how to get it installed. Yes, 'we' know how to do it, or can figure it out pretty easily (with the value of 'we' being pretty much anyone reading this list), but the argument seems to be whether "Joe Average Non-Programmer" could figure it out, or should even have to. A simple: # super-duper-python-thing-just-like-cpan-only-better -i Twisted Should do it, and it should find the proper mirror/source repository itself without further operator intervention or annoyance. At least that's what I get from all this... Steve aka/S aka/ssteinerX From ben+python at benfinney.id.au Thu Dec 24 06:12:12 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 24 Dec 2009 16:12:12 +1100 Subject: [Distutils] Python people want CPAN and how the latter came about References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> Message-ID: <87637x7y6r.fsf@benfinney.id.au> exarkun at twistedmatrix.com writes: > What's with the interest in having packages hosted on PyPI? Economies of scale. There are already numerous tools that work (to greater or lesser degree) with PyPI, and the developers of PyPI have the burden of compatibility with published interfaces. That gives much greater confidence that access to new versions of a distribution will remain accessible in the same way as thousands of other packages, which in turn greatly reduces the cognitive and manual load on down-stream projects working with those distributions. As a small example: with PyPI, I already know how to set up a Debian package that will point to a specific uploaded version of the pristine source, and can then have confidence the same URL will always refer to that same version. The same system then ensures that newer versions will be in a predictable location when they are uploaded. Having already figured that out for one code base hosted on PyPI, I don't have to figure it out any more for other code bases also hosted at PyPI. That is a big attractor of a consistent package repository with a rich standard interface. -- \ ?No smoothen the lion.? ?lion cage, zoo, Czech Republic | `\ | _o__) | Ben Finney From regebro at gmail.com Thu Dec 24 07:22:09 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 07:22:09 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> Message-ID: <319e029f0912232222g3a8c2128r7b7949768f1a4359@mail.gmail.com> On Wed, Dec 23, 2009 at 23:20, Glyph Lefkowitz wrote: > If someone would like to give us a good reason to upload Not having multiple points of failure. When you need to set up an environment for testing or production with all the parts you need, the more servers you need to download modules from, the higher the likelyhood that one of them is down, preventing you from setting up that environment. > or explain how uploading might be made easy, maybe we'd start doing it :). Can't help you there, sorry. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Thu Dec 24 07:42:31 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 07:42:31 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B32997A.3070407@activestate.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B32997A.3070407@activestate.com> Message-ID: <319e029f0912232242uc02eef2i8c261783e1622e8c@mail.gmail.com> On Wed, Dec 23, 2009 at 23:28, Sridhar Ratnakumar wrote: > I suggested PyPI to disallow mere project listings (without sources) and > require sources to be stored in the server. One way to achieve this is > requiring package authors to use the `sdist upload` toolchain Which only means the packages who now is not uploaded wouldn't even be listed on PyPI, which is not an improvement. > While the specific case mentioned above (metadata for a specific or the > latest version of a package) uses HTTP GET and XML, generally speaking .. to > get a) the list of recently releases, b) list of all versions of a package, > one has to use the XmlRpc API methods `changelog` and `package_releases` > respectively. Well, maybe pure http versions of those would help, but on the other hand, if you automate it, why not use xml-rpc? > As often as the mirror sites would update their content (i.e., one or more > times a day). I meant that most of the third-party apps would only need the metadata, or? I might be wrong, I haven't written any yet. :-) The automated documentation that was discussed would only need the source packages. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Thu Dec 24 08:02:47 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 08:02:47 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> References: <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> Message-ID: <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> On Thu, Dec 24, 2009 at 06:06, ssteinerX at gmail.com wrote: > Right now, installing e.g. Twisted, requires finding the website, figuring out which exact file to download, then figuring out exactly how to get it installed. Nope. > # super-duper-python-thing-just-like-cpan-only-better -i Twisted > > Should do it $ /opt/python26/bin/virtualenv twist New python executable in twist/bin/python Installing setuptools............done. $ cd twist/ $ bin/easy_install Twisted Searching for Twisted Reading http://pypi.python.org/simple/Twisted/ Reading http://twistedmatrix.com/ Reading http://www.twistedmatrix.com Reading http://twistedmatrix.com/products/download Reading http://twistedmatrix.com/projects/core/ Best match: Twisted 9.0.0 Downloading http://tmrc.mit.edu/mirror/twisted/Twisted/9.0/Twisted-9.0.0.tar.bz2#md5=93fc2756a09ffd1350c046cc940e4311 Processing Twisted-9.0.0.tar.bz2 [Loads of C-warnings and other stuff removed] Processing dependencies for Twisted Searching for zope.interface Reading http://pypi.python.org/simple/zope.interface/ Best match: zope.interface 3.5.3 Downloading http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.5.3.tar.gz#md5=1fdb9a77f92d3ada3e795a8c9b58d0c6 Processing zope.interface-3.5.3.tar.gz Running zope.interface-3.5.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-uz41Wh/zope.interface-3.5.3/egg-dist-tmp-t5Pzjl Adding zope.interface 3.5.3 to easy-install.pth file Installed /tmp/twist/lib/python2.6/site-packages/zope.interface-3.5.3-py2.6-linux-i686.egg Finished processing dependencies for Twisted Done! > At least that's what I get from all this... We have *that*. Two versions of it even, as pip would likely work as well. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Thu Dec 24 08:05:09 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 08:05:09 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <5b8d13220912211707i5da11b7foc1ac1b86eaf43566@mail.gmail.com> <319e029f0912220002j633a9072l66767105ab5de142@mail.gmail.com> <5b8d13220912220011t3cb7bb4fy8d368a6e0e411f63@mail.gmail.com> <319e029f0912221004g7aa353b9mdbdd8b09e2a30484@mail.gmail.com> <319e029f0912221601o4dac1df4kff1414fef26ec3dc@mail.gmail.com> <51c331ec3db84cceb8f2a8cf911a9aac@preisshare.net> <319e029f0912230108q274ab3cbr9015d2ac4cbbd121@mail.gmail.com> Message-ID: <319e029f0912232305g58248e3cn3493bb3a529bcc19@mail.gmail.com> On Thu, Dec 24, 2009 at 00:10, David Lyon wrote: > On Wed, 23 Dec 2009 10:08:24 +0100, Lennart Regebro > wrote: >>> There is no .tar.gz, .zip, .bz2, .exe, .msi or .egg concept of >>> packages in perl. And having to pick one.. that may or may not >>> be right for your configuration. >> >> So your usecase, that a Windows user refuses to install anything else >> than .msi files is solved in CPAN by the user not installing using >> CPAN. Well, that's handy. And the Python situation is worse how, >> exactly? > > The same operations take a longer time to complete than the > equivalent operations do with perl. There is no "same operation". Your answer has no connection to what you answered, and hence make no sense. >> That is the most ridicolous excuse I've heard in a long time. People >> are saying to Guido that "Python needs CPAN" and wonders what that >> means. You come with you usual anti distutils rant, I ask *you* what >> the connection is, and you tell med to go hassle Guido. > > Yeah, I did. or maybe you can google on it.. And you still don't understand how stupid that is. QED. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Thu Dec 24 08:08:24 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 08:08:24 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> Message-ID: <319e029f0912232308x6e1a953bj8424c7833f00b313@mail.gmail.com> On Wed, Dec 23, 2009 at 23:32, Tarek Ziad? wrote: > I could even add a --dist-file option to it so you can point an > existing archive to push at PyPI, > so running sdist or another *dist command wouldn't be mandatory anymore I think that would be an excellent idea. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Thu Dec 24 09:22:10 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Dec 2009 09:22:10 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <94bdd2610912210314r450f8857i322dd60a11deb870@mail.gmail.com> <94bdd2610912210318o2fdf1e19qf325921b512c2310@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> <4B3279E9.4010507@v.loewis.de> <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> Message-ID: <4B3324B2.3030708@v.loewis.de> > Some applications doesn't care about the micro version, but will care about > the minor version because those can introduce API and syntax changes. > > e.g. If at some point I have a Python 2.6 program that uses the "with" > keyword, and I don't want to introduce a __future__ import in my code, > I'll just use : > > Requires-Python: 2.6 > > And I will not care much about the micro version in that case, since > Python will not change its syntax/API in a micro release. I think that's a mistake. Specifying "2.6" is equivalent to specifying "==2.6". With your proposed meaning of "==", either == is not transitive, or all versions compare equal. As 2.6==2.6.1 and 2.6==2.6.2, we get (with transitive ==), 2.6.1==2.6.2. Furthermore, if the same wildcarding applies to major versions as well, i.e. Requires-Python: 3 means any 3.x, then we have 3==3.1, 3==3.2, 3.1==3.1.3, 3.2==3.2.4, and, transitively, 3.1.3==3.2.4. This is undesirable. >>>> Does that apply to all PEP 386 version numbers? >>> Not sure what you mean here, >> If I specify >> >> Requires-Dist: zope.interface (3.6) >> >> and I have zope.interface 3.6.1b4, is the requirement satisfied? >> What if I have 3.6b4: is it then satisfied? > > 3.6 would include all 3.6.x releases as well. So 3.6b4 is excluded > since it does not belong to the 3.6 series, but 3.6.1b4 is included. Please define "belongs to the 3.6 series". In PEP 386 terminology, I would expect that this means "the 'version' part is 3.6", so 3.6b4 *does* belong to the 3.6 series. Regards, Martin From martin at v.loewis.de Thu Dec 24 09:33:50 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Dec 2009 09:33:50 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B32894B.8090109@activestate.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> Message-ID: <4B33276E.2070900@v.loewis.de> > 1/ Missing packages (eg: Twisted is not there); which is why > easy_install/pip had to resolve to scrapping project webpages for > guessing download links. In CPAN, almost all module authors upload their > sources via PAUSE. How do you propose to change that? I think it should be the choice of the package authors whether they upload their software to the central repository, or to their own home page. > 2/ No metadata: When only source tarballs are stored > [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to > a) get the source for latest version, Extract a version number from each file name, and sort the versions, then use the largest (which is 0.9.7 at the moment). > b) get the source for a particular version? Put the version number into the file name, and access the resulting file. > The former is more of a community issue. Often Python package authors > are not using `sdist upload` (whereas this seems to be the convention in > the Perl world). My guess is that this is enforced by the tools. If they don't upload to PAUSE, CPAN.pm won't be able to download it. Now, you are free to build a tool that enforces the same restriction. I would doubt that people would use it, since it couldn't install many packages. > What this means is that PyPI has to serve the purpose of being a central > package repository (like CPAN) by a) disallowing mere listings (without > sources) and requiring sources to be stored in the server, b) storing > the metadata along with the sources (so anyone processing it wouldn't > have to extract the source and rely on a PKG-INFO file - which may or > may not exist). If you want to retrieve the metadata for a specific version without XML-RPC, you can access http://pypi.python.org/pypi?:action=doap&name=Pylons&version=0.9.7 Regards, Martin From martin at v.loewis.de Thu Dec 24 09:35:17 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Dec 2009 09:35:17 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B328C0D.7080705@activestate.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B328C0D.7080705@activestate.com> Message-ID: <4B3327C5.5000405@v.loewis.de> > 3/ Indices such as http://www.cpan.org/modules/01modules.mtime.html (or > TXT files) to get a) recently released packages, b) list of release > versions for a particular package, and so on (which would obviate the > XmlRpc interface) That is available as http://pypi.python.org/pypi?%3Aaction=rss Regards, Martin From martin at v.loewis.de Thu Dec 24 09:40:42 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Dec 2009 09:40:42 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B3298D7.6040706@cryptelium.net> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> Message-ID: <4B33290A.80809@v.loewis.de> > The tiny 10MB upload is a blocker i think also. > For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. The limit is 20MB now. If you need a larger limit let me know. Unfortunately, I cannot find out what the size of minitage.paste.extras is, as their download URL (http://distfiles.minitage.org/public/externals/minitage/) seems broken. Regards, Martin From martin at v.loewis.de Thu Dec 24 09:54:16 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 24 Dec 2009 09:54:16 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> Message-ID: <4B332C38.1080002@v.loewis.de> > What's with the interest in having packages hosted on PyPI? "Because it would then be more like CPAN". Some people claim that one key of CPAN's success is that it has all files stored in the archive, and that it then allows mirroring with rsync and ftp. They claim that without that, PyPI can't be simple, and without that, the requirement "Python needs CPAN" can't be met. > I'm not specifically opposed to this, but I don't see any reason it > would benefit anyone either. It'd be awesome if someone could explain > this. Perhaps if the answer were included somewhere on PyPI or in the > distutils documentation, more people would see the light and upload > their packages. For all use cases except of mirroring, I don't see the need to upload, either. For mirroring, it would be easier to write mirroring tools if all packages uploaded their code (assuming you want to end up with a complete off-line mirror). Regards, Martin From martin at v.loewis.de Thu Dec 24 10:02:17 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 24 Dec 2009 10:02:17 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <4B329EE6.1000801@activestate.com> <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> Message-ID: <4B332E19.2090507@v.loewis.de> > A separate issue with "setup.py upload", though, is that it really wants > one of two undesirable things: > > * the upload is done at the same time as the release package is generated > * the release package is generated twice > > The former requires that proper credentials are available to whoever is > creating the release package. Historically for Twisted, this isn't how > things have been set up. We could probably deal with it, but it would > be nice if it were not a requirement. It actually isn't. Upload will upload all files that are listed on distribution.dist_files, which is a three-tuple of (command/filetype, pyversion, filename). So if you somehow (e.g. with a new command, or hard-coded) put stuff into dist_files, you could arrange for python setup.py pickup_files upload to upload the pre-built files; thereby you can upload files as source that had not been generated by sdist. Regards, Martin From ben+python at benfinney.id.au Thu Dec 24 10:25:39 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 24 Dec 2009 20:25:39 +1100 Subject: [Distutils] Python people want CPAN and how the latter came about References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> Message-ID: <87oclo7mgc.fsf@benfinney.id.au> "Martin v. L?wis" writes: > I think it should be the choice of the package authors whether they > upload their software to the central repository, or to their own home > page. Why do you think that should continue? Some of the costs of that inconsistency have already been described in this thread. What are the benefits to PyPI users of this inconsistency, and are we sure that the benefits outweigh the costs? -- \ ?I got contacts, but I only need them when I read, so I got | `\ flip-ups.? ?Steven Wright | _o__) | Ben Finney From ziade.tarek at gmail.com Thu Dec 24 10:32:17 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 10:32:17 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <4B3324B2.3030708@v.loewis.de> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> <4B3279E9.4010507@v.loewis.de> <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> <4B3324B2.3030708@v.loewis.de> Message-ID: <94bdd2610912240132t345b4d8bnc1bc81d5ae82b194@mail.gmail.com> 2009/12/24 "Martin v. L?wis" : >> Some applications doesn't care about the micro version, but will care about >> the minor version because those can introduce API and syntax changes. >> >> e.g. If at some point I have a Python 2.6 program that uses the "with" >> keyword, and I don't want to introduce a __future__ import in my code, >> I'll just use : >> >> Requires-Python: 2.6 >> >> And I will not care much about the micro version in that case, since >> Python will not change its syntax/API in a micro release. > > I think that's a mistake. Specifying "2.6" is equivalent to specifying > "==2.6". With your proposed meaning of "==", either == is not > transitive, or all versions compare equal. As 2.6==2.6.1 and 2.6==2.6.2, > we get (with transitive ==), 2.6.1==2.6.2. > > Furthermore, if the same wildcarding applies to major versions as well, > i.e. > > Requires-Python: 3 > > means any 3.x, then we have 3==3.1, 3==3.2, 3.1==3.1.3, 3.2==3.2.4, > and, transitively, 3.1.3==3.2.4. This is undesirable. Why this is undesirable ? I find it highly desirable for developers that don't want to bother with Python version details, while it will let other developers give precise versions if they need. You'll have to convince me otherwise by explaining why it's a mistake to apply wildcarding when the full version is not provided. [..] >> 3.6 would include all 3.6.x releases as well. So 3.6b4 is excluded >> since it does not belong to the 3.6 series, but 3.6.1b4 is included. > > Please define "belongs to the 3.6 series". In PEP 386 terminology, > I would expect that this means "the 'version' part is 3.6", so > 3.6b4 *does* belong to the 3.6 series. Sorry, here's the definition I've put behind the word belong: A version belongs to the 3.6 series if : 3.6 <= VERSION < 3.7 3.6b4 is the fourth beta release, so : 3.6b4 < 3.6 <= VERSION < 3.7 So obviously, if the developer ask for 3.6, he doesn't want a preview of that version, he wants that version Regards, Tarek From a.cavallo at cavallinux.eu Thu Dec 24 10:28:57 2009 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Thu, 24 Dec 2009 10:28:57 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B332E19.2090507@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <4B329EE6.1000801@activestate.com> <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> <4B332E19.2090507@v.loewis.de> Message-ID: <2BD28EF2-66FE-4A46-B268-E5015D4329F6@cavallinux.eu> Finally somebody had few doubts about CPAN...please have a look ti a "just-posted" article on slashdot. That mess is CPAN was my original reason to discard perl in first (and switching to python): no two installed perl are ever the same. No way to reliably reproduce the same environment and no auditing possible: and this is a requirement in a professional environment. I was (and I'm still) happy the way distutils works: it might be enhanched but it's right in the scope (creating an installer). The way the software is indexed, managed, have the deps tracked and retrieved should not be in that scope: it is a job for something else (a tool). If I had to push for two ideas I need for my work (yes it is done in python): - integration with the system installer and updating mechanisms in place - don't assume internet available Regards, Antonio From kiorky at cryptelium.net Thu Dec 24 10:54:11 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 24 Dec 2009 10:54:11 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B33290A.80809@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> Message-ID: <4B333A43.5030305@cryptelium.net> Martin v. L?wis a ?crit : >> The tiny 10MB upload is a blocker i think also. >> For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. > > The limit is 20MB now. If you need a larger limit let me know. > > Unfortunately, I cannot find out what the size of minitage.paste.extras > is, as their download URL > (http://distfiles.minitage.org/public/externals/minitage/) > seems broken. > Uhm, the server was down a little time earlier. It is fine now, normally. The size of this dist is 12mb. The last i tried to upload it was maybe 8 monthes ago if you changed the limit in the meantime. > Regards, > Martin -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From ziade.tarek at gmail.com Thu Dec 24 11:03:48 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 11:03:48 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B332E19.2090507@v.loewis.de> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <4B329EE6.1000801@activestate.com> <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> <4B332E19.2090507@v.loewis.de> Message-ID: <94bdd2610912240203p5aca0411ue74cbbd3572c9ca1@mail.gmail.com> On Thu, Dec 24, 2009 at 10:02 AM, "Martin v. L?wis" wrote: >> A separate issue with "setup.py upload", though, is that it really wants >> one of two undesirable things: >> >> ?* the upload is done at the same time as the release package is generated >> ?* the release package is generated twice >> >> The former requires that proper credentials are available to whoever is >> creating the release package. ?Historically for Twisted, this isn't how >> things have been set up. ?We could probably deal with it, but it would >> be nice if it were not a requirement. > > It actually isn't. Upload will upload all files that are listed on > distribution.dist_files, which is a three-tuple of (command/filetype, > pyversion, filename). So if you somehow (e.g. with a new command, or > hard-coded) put stuff into dist_files, you could arrange for > > python setup.py pickup_files upload > > to upload the pre-built files; thereby you can upload files as source > that had not been generated by sdist. > That's why I've proposed to add a --dist-file option to the upload command, Tarek From kiorky at cryptelium.net Thu Dec 24 11:06:51 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 24 Dec 2009 11:06:51 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B33290A.80809@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> Message-ID: <4B333D3B.7010403@cryptelium.net> Martin v. L?wis a ?crit : >> The tiny 10MB upload is a blocker i think also. >> For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. > > The limit is 20MB now. If you need a larger limit let me know. Cool! I tested with success the upload. Size of the dist is 12MB. However, i don't like also that much the sdist upload command for big files. It may be an idea to setup/add some ssh+key procedure to upload packages using ssh. Then we could use rsync or such program that can resume upload on errors without having to reupload the whole. Nevertheless, i don't know what's worse, because i don't think there are that much big files to upload. In the packages i maintain, there is only minitage.paste.extras concerned for example. Maybe some started would be to find packages on pypi without dists and verify on their relative download page the bigest size we can find out there. > Unfortunately, I cannot find out what the size of minitage.paste.extras > is, as their download URL > (http://distfiles.minitage.org/public/externals/minitage/) > seems broken. > Server was down a little earlier, /me was sleeping. It's fine now. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From ziade.tarek at gmail.com Thu Dec 24 11:06:38 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 11:06:38 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> References: <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <4B329EE6.1000801@activestate.com> <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> Message-ID: <94bdd2610912240206if99916fn9bdbd1442144419a@mail.gmail.com> 2009/12/24 : [..] > > Release packages for Twisted are constructed using some extra file- finding > logic that sdist doesn't provide. ?Additionally, for years distutils was > seen as a blind alley, so we didn't bother to try to create a > distutils-friendly substitute. ?Partially because it seems that distutils > has turned a corner over the last year, we are actually (slowly, with > difficulty) working towards a more distutils-integrated solution (we're > going to try to override sdist with a command based on our existing custom > file-finding code). ?This may result in something we can use with "setup.py > upload" (but this isn't currently the primary goal, it would just be a happy > side effect). That's quite a good news. Let me know if I can help somehow in that process. In any case I am interested in the problems you've had with sdist in order to improve it. Wether this happens in Distutils or in Distribute if it's too late for 2.7 Tarek From kiorky at cryptelium.net Thu Dec 24 11:08:12 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 24 Dec 2009 11:08:12 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <87oclo7mgc.fsf@benfinney.id.au> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> Message-ID: <4B333D8C.9010601@cryptelium.net> Ben Finney a ?crit : > "Martin v. L?wis" writes: > >> I think it should be the choice of the package authors whether they >> upload their software to the central repository, or to their own home >> page. > > Why do you think that should continue? Some of the costs of that > inconsistency have already been described in this thread. What are the Totally agree, all must be on pypi. > benefits to PyPI users of this inconsistency, and are we sure that the > benefits outweigh the costs? > -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From ziade.tarek at gmail.com Thu Dec 24 11:14:56 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 11:14:56 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B333D3B.7010403@cryptelium.net> References: <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> <4B333D3B.7010403@cryptelium.net> Message-ID: <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> On Thu, Dec 24, 2009 at 11:06 AM, kiorky wrote: > > > Martin v. L?wis a ?crit : >>> The tiny 10MB upload is a blocker i think also. >>> For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. >> >> The limit is 20MB now. If you need a larger limit let me know. > > Cool! > I tested with success the upload. > Size of the dist is 12MB. > > However, i don't like also that much the sdist upload command for big files. > It may be an idea to setup/add some ssh+key procedure to upload packages using ssh. > Then we could use rsync or such program that can resume upload on errors without > having to reupload the whole. I don't think it worth the pain, with the speed of nowadays connections. But I could add a curl upload progress bar in the upload command. If you think this is useful let me know. That said, some people have expressed the desire to be able to interact with PyPI through SSH so they could drop the basic authentication and use their keys when registering/uploading. But that was not related to the size of files. Tarek -- Tarek Ziad? | http://ziade.org From kiorky at cryptelium.net Thu Dec 24 11:20:24 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 24 Dec 2009 11:20:24 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> References: <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> <4B333D3B.7010403@cryptelium.net> <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> Message-ID: <4B334068.2060900@cryptelium.net> Tarek Ziad? a ?crit : > On Thu, Dec 24, 2009 at 11:06 AM, kiorky wrote: >> >> Martin v. L?wis a ?crit : > I don't think it worth the pain, with the speed of nowadays > connections. But I could add a curl upload progress bar in the upload > command. If you think this is useful let me know. No. progress is useless but indeed beautiful and a nice enlightement. If it's a 2 minutes dev, it can be nice, on the other hand i think we have much problems to solve :p. The only feature i thought useful is partial upload (to resume on errors). We, in France, have for the whole good connections but not everyone. I have a quite poor connection here for example :). > > That said, some people have expressed the desire to be able to > interact with PyPI through SSH so they could drop the basic > authentication and use their keys when registering/uploading. > But that was not related to the size of files. > Both i think, it's easier to use the transfer programs that we are used to use to transfer things. And ssh is damn good for that, easy to setup, secure and so on. But it's only butter, in fact i am just happy with sdist upload. > Tarek -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From martin at v.loewis.de Thu Dec 24 11:29:07 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 24 Dec 2009 11:29:07 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <87oclo7mgc.fsf@benfinney.id.au> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> Message-ID: <4B334273.1050707@v.loewis.de> >> I think it should be the choice of the package authors whether they >> upload their software to the central repository, or to their own home >> page. > > Why do you think that should continue? Some of the costs of that > inconsistency have already been described in this thread. What are the > benefits to PyPI users of this inconsistency, and are we sure that the > benefits outweigh the costs? The benefits are not to the package users, clearly. Instead, they are to the package authors, which don't have to change their release processes (as also described in this thread). Regards, Martin From ziade.tarek at gmail.com Thu Dec 24 11:44:11 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 11:44:11 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B334068.2060900@cryptelium.net> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> <4B333D3B.7010403@cryptelium.net> <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> <4B334068.2060900@cryptelium.net> Message-ID: <94bdd2610912240244q385fc4ebg7d86aeb19610afbe@mail.gmail.com> On Thu, Dec 24, 2009 at 11:20 AM, kiorky wrote: [..] >> That said, some people have expressed the desire to be able to >> interact with PyPI through SSH so they could drop the basic >> authentication and use their keys when registering/uploading. >> But that was not related to the size of files. >> > > Both i think, it's easier to use the transfer programs that we are used to use > to transfer things. > And ssh is damn good for that, easy to setup, secure and so on. > > But it's only butter, in fact i am just happy with sdist upload. Although SSH is quite a heavy development on PyPI side, it means we would have to implement an SSH server. (like Zope did I think for their development server, using Paramiko IIRC) From regebro at gmail.com Thu Dec 24 11:44:27 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 11:44:27 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <87oclo7mgc.fsf@benfinney.id.au> References: <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> Message-ID: <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> On Thu, Dec 24, 2009 at 10:25, Ben Finney wrote: > "Martin v. L?wis" writes: > >> I think it should be the choice of the package authors whether they >> upload their software to the central repository, or to their own home >> page. > > Why do you think that should continue? Because otherwise you can't register packages in PyPI without uploading them, and that means that those who do not want to upload them also will not register them. Forcing people to do what you think they should do will not make them make more or better work. It will just make them do *less* work. It's better to figure out why some people don't upload the packages, and if they have a reasonable reason, fix it. -- Lennart Regebro: http://regebro.wordpress.com/ I thought I could organize freedom. How Scandinavian of me. --Bj?rk +33 661 58 14 64 From martin at v.loewis.de Thu Dec 24 11:44:54 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Dec 2009 11:44:54 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912240132t345b4d8bnc1bc81d5ae82b194@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <4B301D54.6050109@v.loewis.de> <94bdd2610912230250u2c0b7695n1be4ec4ae3c2c41b@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> <4B3279E9.4010507@v.loewis.de> <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> <4B3324B2.3030708@v.loewis.de> <94bdd2610912240132t345b4d8bnc1bc81d5ae82b194@mail.gmail.com> Message-ID: <4B334626.6040505@v.loewis.de> >>> And I will not care much about the micro version in that case, since >>> Python will not change its syntax/API in a micro release. >> I think that's a mistake. Specifying "2.6" is equivalent to specifying >> "==2.6". With your proposed meaning of "==", either == is not >> transitive, or all versions compare equal. As 2.6==2.6.1 and 2.6==2.6.2, >> we get (with transitive ==), 2.6.1==2.6.2. >> >> Furthermore, if the same wildcarding applies to major versions as well, >> i.e. >> >> Requires-Python: 3 >> >> means any 3.x, then we have 3==3.1, 3==3.2, 3.1==3.1.3, 3.2==3.2.4, >> and, transitively, 3.1.3==3.2.4. This is undesirable. > > Why this is undesirable ? I find it undesirable if 3.1.3==3.2.4. Then, if I specify Requires-Python: 3.1.3 and I have 3.2.4 installed, it would accept that (because the version numbers are "equal", i.e. "=="). > I find it highly desirable for developers that don't want to > bother with Python version details, while it will let other developers > give precise versions if they need. That is desirable, indeed. However, it is also possible without == performing some wildcard matching: Requires-Python: >=2.5,<2.6 will also specify that 2.5.x is required. > You'll have to convince me otherwise by explaining why it's a mistake > to apply wildcarding when the full version is not provided. See above: is == transitive or not? I think it is counter-intuitive if the == operator on versions does anything else than comparing version strings for string equality. If you think that a range matching operator is needed, perhaps a different symbol could be used (such as ~=)? >>> 3.6 would include all 3.6.x releases as well. So 3.6b4 is excluded >>> since it does not belong to the 3.6 series, but 3.6.1b4 is included. >> Please define "belongs to the 3.6 series". In PEP 386 terminology, >> I would expect that this means "the 'version' part is 3.6", so >> 3.6b4 *does* belong to the 3.6 series. > > Sorry, here's the definition I've put behind the word belong: > > A version belongs to the 3.6 series if : 3.6 <= VERSION < 3.7 > > 3.6b4 is the fourth beta release, so : > > 3.6b4 < 3.6 <= VERSION < 3.7 > > So obviously, if the developer ask for 3.6, he doesn't want a preview of that > version, he wants that version Where is that specified? I can't find it in the PEP. If you really want this == operator, please add a Discussion section to the PEP discussing that there was objection to this operator, proposing that it should do string== instead, and that this specific semantics was chosen because . Regards, Martin P.S. I notice that your == definition and my explicit rewrite of it have the same flaw: in your example, "3.7b1" would also match a version specification of "3.6". I.e. prereleases of the next releases still fall into the previous range. From martin at v.loewis.de Thu Dec 24 11:46:13 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Dec 2009 11:46:13 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912240203p5aca0411ue74cbbd3572c9ca1@mail.gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <4B329EE6.1000801@activestate.com> <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> <4B332E19.2090507@v.loewis.de> <94bdd2610912240203p5aca0411ue74cbbd3572c9ca1@mail.gmail.com> Message-ID: <4B334675.5070009@v.loewis.de> >> python setup.py pickup_files upload >> >> to upload the pre-built files; thereby you can upload files as source >> that had not been generated by sdist. >> > > That's why I've proposed to add a --dist-file option to the upload command, The tricky thing may be to find out what kind of file that is, so that option would somehow need two parameters. Regards, Martin From python at davidrobins.net Thu Dec 24 11:46:59 2009 From: python at davidrobins.net (David Robins) Date: Thu, 24 Dec 2009 02:46:59 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> Message-ID: <20091224104658.GA12434@rivendell.internal> On Thu, Dec 24, 2009 at 02:59:28AM -0000, exarkun at twistedmatrix.com wrote: > There have been a few responses to Glyph's mention that "setup.py > upload" doesn't work for Twisted. I'm much more curious to hear replies > to his other point - "nobody cares anyway". > > What's with the interest in having packages hosted on PyPI? > > I'm not specifically opposed to this, but I don't see any reason it > would benefit anyone either. It'd be awesome if someone could explain > this. Perhaps if the answer were included somewhere on PyPI or in the > distutils documentation, more people would see the light and upload > their packages. > > Jean-Paul Some reasons to have PyPI host packages have already been mentioned in this thread: it makes mirroring easier, and it makes it easier for individuals to build new services (web sites primarily) that present new interfaces to the Python package collection. Mirroring for its own sake is some use, but being able to grab the entire Python package repository easily from a single source is valuable for the second goal, that of furnishing the foundation ("shoulders of giants" and all that) for those with vision (and round tuits) to take the next step. If I wanted to host a site that (e.g.) indexed Python modules from PyPI by module (not package) name, and extracted and provided the documentation in HTML format, from what I've been reading I'd have to build a scraper or XMLRPC tool to walk PyPI, and then for each package, download it from another site (that may not have the uptime or scalability of PyPI), a nontrivial burden on aspiring visionaries that just want to build an addition and then go have a beer and discuss further improvements. For CPAN, as others have said already, a recursive FTP or rsync would do the job leaving people free to spend their time innovating. (As a point of practical interest, what _would_ be the most efficient way to download the entire set of Python modules listed on PyPI? A search comes up with z3c.pypimirror, http://pypi.python.org/pypi/z3c.pypimirror; is this the standard tool?) It's about affordability, in the sense Donald Norman uses it in "The Design of Everyday Things." Does PyPI afford extension? Should it? Having "one way to do it" is Pythonic; so is having several people rush off and build new things sufficiently chaotic as to be unpythonic? Arguably no, because they would be prototypes - brainstorming - and the community can synthesize the best of each (hey, if "TMTOWTDI" Perl folk can do it...) and reintegrate them as necessary and proper. From martin at v.loewis.de Thu Dec 24 11:51:30 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Dec 2009 11:51:30 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912240244q385fc4ebg7d86aeb19610afbe@mail.gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> <4B333D3B.7010403@cryptelium.net> <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> <4B334068.2060900@cryptelium.net> <94bdd2610912240244q385fc4ebg7d86aeb19610afbe@mail.gmail.com> Message-ID: <4B3347B2.8040204@v.loewis.de> > Although SSH is quite a heavy development on PyPI side, it means we > would have to implement > an SSH server. (like Zope did I think for their development server, > using Paramiko IIRC) Not necessarily: it might be possible to use sshd. Regards, Martin From ziade.tarek at gmail.com Thu Dec 24 11:51:50 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 11:51:50 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B334675.5070009@v.loewis.de> References: <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <4B329EE6.1000801@activestate.com> <20091224031155.15596.739933290.divmod.xquotient.525@localhost.localdomain> <4B332E19.2090507@v.loewis.de> <94bdd2610912240203p5aca0411ue74cbbd3572c9ca1@mail.gmail.com> <4B334675.5070009@v.loewis.de> Message-ID: <94bdd2610912240251p247fe2cn5daf5ebd6ce6194@mail.gmail.com> 2009/12/24 "Martin v. L?wis" : >>> python setup.py pickup_files upload >>> >>> to upload the pre-built files; thereby you can upload files as source >>> that had not been generated by sdist. >>> >> >> That's why I've proposed to add a --dist-file option to the upload command, > > The tricky thing may be to find out what kind of file that is, so that > option would somehow need two parameters. Yes. I was considering something in these lines: $ python setup.py upload --dist-file=sdist:/path/to/archive.tgz where the files are suffixed by their format. But that's just a first thought. > > Regards, > Martin > -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Thu Dec 24 12:00:05 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Dec 2009 12:00:05 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <20091224104658.GA12434@rivendell.internal> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <20091224104658.GA12434@rivendell.internal> Message-ID: <4B3349B5.3000804@v.loewis.de> > Some reasons to have PyPI host packages have already been mentioned in > this thread: it makes mirroring easier, and it makes it easier for > individuals to build new services (web sites primarily) that present new > interfaces to the Python package collection. Mirroring for its own sake > is some use, but being able to grab the entire Python package repository > easily from a single source is valuable for the second goal, that of > furnishing the foundation ("shoulders of giants" and all that) for those > with vision (and round tuits) to take the next step. That is fairly easily possible today, even without everybody uploading all files. It isn't easy *per se*, but needs a lot of code. However, this code has already been written, and using it is fairly easy. > If I wanted to host a site that (e.g.) indexed Python modules from PyPI > by module (not package) name, and extracted and provided the > documentation in HTML format, from what I've been reading I'd have to > build a scraper or XMLRPC tool to walk PyPI, and then for each package, > download it from another site (that may not have the uptime or > scalability of PyPI), a nontrivial burden on aspiring visionaries that > just want to build an addition and then go have a beer and discuss > further improvements. Not at all. You would just use one of the ten or so packages that already do precisely that, and use it. > (As a point of practical interest, what _would_ be the most efficient > way to download the entire set of Python modules listed on PyPI? A > search comes up with z3c.pypimirror, > http://pypi.python.org/pypi/z3c.pypimirror; is this the standard tool?) There are a number of other mirroring tools, such as EggBasket and collective.eggproxy. For mirroring the whole index, pypimirror is probably the best starting point. Regards, Martin From ziade.tarek at gmail.com Thu Dec 24 12:15:58 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 12:15:58 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B3349B5.3000804@v.loewis.de> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <20091224104658.GA12434@rivendell.internal> <4B3349B5.3000804@v.loewis.de> Message-ID: <94bdd2610912240315l70b2e3ecwd95abe840a3d1d98@mail.gmail.com> On Thu, Dec 24, 2009 at 12:00 PM, "Martin v. L?wis" wrote: [..] > > There are a number of other mirroring tools, such as EggBasket and > collective.eggproxy. For mirroring the whole index, pypimirror is > probably the best starting point. collective.eggproxy is particular though: it's a proxy-cache on PyPI (on any other PyPI-like server), that can be used by developers to collect in a local cache the archives they have downloaded at least once, so they don't call PyPI anymore. The tree structure is downloaded but is filled only on requests. IOW it gets filled when users uses it as their PyPI index. In tools like buildout for instance, and they end up with a local subset of the archive they *really* use and need. We did that in my previous company because we didn't want to set up and maintain a 7 giga mirror, while we just used probably less than 5% of PyPI archives. The caveat is that when a new version is up, it is downloaded only when someone claims for it, unlike mirrors that gets updated in crons. But it's not really a big deal because my developers team back then was updating their buildouts more often that any mirror cron out there I am pretty sure ;) Tarek From kiorky at cryptelium.net Thu Dec 24 12:24:28 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 24 Dec 2009 12:24:28 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B3347B2.8040204@v.loewis.de> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> <4B333D3B.7010403@cryptelium.net> <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> <4B334068.2060900@cryptelium.net> <94bdd2610912240244q385fc4ebg7d86aeb19610afbe@mail.gmail.com> <4B3347B2.8040204@v.loewis.de> Message-ID: <4B334F6C.9040603@cryptelium.net> Martin v. L?wis a ?crit : >> Although SSH is quite a heavy development on PyPI side, it means we >> would have to implement >> an SSH server. (like Zope did I think for their development server, >> using Paramiko IIRC) > > Not necessarily: it might be possible to use sshd. Thas was my underlying though too. > > Regards, > Martin > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From kiorky at cryptelium.net Thu Dec 24 12:28:08 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 24 Dec 2009 12:28:08 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> References: <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> Message-ID: <4B335048.4060707@cryptelium.net> Lennart Regebro a ?crit : > On Thu, Dec 24, 2009 at 10:25, Ben Finney wrote: >> "Martin v. L?wis" writes: > Because otherwise you can't register packages in PyPI without > uploading them, and that means that those who do not want to upload > them also will not register them. > Forcing people to do what you think they should do will not make them Why forcing them ? How about a cronjob or something like that which find packages without distribution and get from there all distributions possible on their relative homepages and mirror them on Pypi ? It's not perfect, but it s an idea. One of the drawback is that Pypi can have outdated things. It would be also a big help in the meantime packagers update their release procedure. > make more or better work. It will just make them do *less* work. It's > better to figure out why some people don't upload the packages, and if > they have a reasonable reason, fix it. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From ben+python at benfinney.id.au Thu Dec 24 13:17:58 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 24 Dec 2009 23:17:58 +1100 Subject: [Distutils] Python people want CPAN and how the latter came about References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <4B334273.1050707@v.loewis.de> Message-ID: <87k4wc7eh5.fsf@benfinney.id.au> "Martin v. L?wis" writes: > The benefits are not to the package users, clearly. That should immediately make us suspicious, then. If PyPI is for anything, surely it is for the package recipients *more than* for package developers. > Instead, they are to the package authors, which don't have to change > their release processes (as also described in this thread). The needs of package developers are important, of course; but, I argue, always in the service of making packages available to recipients. Lennart Regebro writes: > Forcing people to do what you think they should do will not make them > make more or better work. It will just make them do *less* work. That isn't a good argument. By the same logic, PyPI should not reject *any* upload, to avoid ?forcing? uploaders to do extra work. No, PyPI needs to reject uploads on some criteria; those criteria will necessarily involve work (whether manual or automated) on the part of uploaders. This discussion is about what those criteria should be. -- \ ?Why was I with her? She reminds me of you. In fact, she | `\ reminds me more of you than you do!? ?Groucho Marx | _o__) | Ben Finney From martin at v.loewis.de Thu Dec 24 14:30:48 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 24 Dec 2009 14:30:48 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <87k4wc7eh5.fsf@benfinney.id.au> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <4B334273.1050707@v.loewis.de> <87k4wc7eh5.fsf@benfinney.id.au> Message-ID: <4B336D08.8060308@v.loewis.de> Ben Finney wrote: > "Martin v. L?wis" writes: > >> The benefits are not to the package users, clearly. > > That should immediately make us suspicious, then. > > If PyPI is for anything, surely it is for the package recipients *more > than* for package developers. I thought so when adding voting and comments to PyPI. Boy was I wrong. > That isn't a good argument. By the same logic, PyPI should not reject > *any* upload, to avoid ?forcing? uploaders to do extra work. PyPI's rejection of certain uploads is primarily to prevent spam from being uploaded. It has nothing to do with setting quality standards of some kind. Regards, Martin From ssteinerx at gmail.com Thu Dec 24 14:44:52 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 24 Dec 2009 08:44:52 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> References: <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> Message-ID: <7CB35A3D-43E4-4FF4-ABC7-7D559B7A47F0@gmail.com> On Dec 24, 2009, at 2:02 AM, Lennart Regebro wrote: > On Thu, Dec 24, 2009 at 06:06, ssteinerX at gmail.com wrote: >> Right now, installing e.g. Twisted, requires finding the website, figuring out which exact file to download, then figuring out exactly how to get it installed. > > Nope. > >> # super-duper-python-thing-just-like-cpan-only-better -i Twisted >> >> Should do it > > $ /opt/python26/bin/virtualenv twist [snip] Ok, so easy_install works for Twisted. Yay. > Installed /tmp/twist/lib/python2.6/site-packages/zope.interface-3.5.3-py2.6-linux-i686.egg > Finished processing dependencies for Twisted > > Done! > >> At least that's what I get from all this... > > We have *that*. Two versions of it even, as pip would likely work as well. Ok, there's easy_install, the direct download route, and pip. I think that's exactly *not*: # super-duper-python-thing-just-like-cpan-only-better -i Twisted And, I picked Twisted 'cause it was already in the discussion. There are at least three ways to install it and I'm really concerned that what you just seemed to suggest was that the much-maligned-and-soon-to-be-deprecated `easy_install` is the `super-duper-python-thing-just-like-cpan-only-better` of today? S From regebro at gmail.com Thu Dec 24 14:47:10 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 14:47:10 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <20091224104658.GA12434@rivendell.internal> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <20091224104658.GA12434@rivendell.internal> Message-ID: <319e029f0912240547y545a61bcmcc8bab74967fa9c6@mail.gmail.com> On Thu, Dec 24, 2009 at 11:46, David Robins wrote: > Some reasons to have PyPI host packages have already been mentioned in > this thread: it makes mirroring easier Uhm, mirroring PyPI, no. Because those packages aren't there. Setting up your own package server, yes. It's not exactly the same thing. Plone has set up their own mirroring of all the packages needed to set up Plone, to avoid having problems with downed servers etc. That's not a PyPI mirror, it's something else. PyPI is not a vague set of every package that exists in CPAN seems to be in some peoples view of the world. It's an index of packages. You can also host them there. Some people choose not to. That's up to them, and should continue to be up to them. Making it easier to actually find the package would probably be a good thing, such as including the URL's for all the files in the metadata for a package, or similar. Forcing everybody to upload to PyPI is not the correct road to travel. > and it makes it easier for > individuals to build new services (web sites primarily) that present new > interfaces to the Python package collection. Why? They can simply ignore packages that aren't uploaded. That would increase the incentive to upload. No need to force peoples hand. Freedom baby, yeah! -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From ziade.tarek at gmail.com Thu Dec 24 14:55:58 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 24 Dec 2009 14:55:58 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912240547y545a61bcmcc8bab74967fa9c6@mail.gmail.com> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <20091224104658.GA12434@rivendell.internal> <319e029f0912240547y545a61bcmcc8bab74967fa9c6@mail.gmail.com> Message-ID: <94bdd2610912240555m32f5c8e3i9c46da85f6995b3b@mail.gmail.com> On Thu, Dec 24, 2009 at 2:47 PM, Lennart Regebro wrote: > On Thu, Dec 24, 2009 at 11:46, David Robins wrote: >> Some reasons to have PyPI host packages have already been mentioned in >> this thread: it makes mirroring easier > > Uhm, mirroring PyPI, no. Because those packages aren't there. Setting > up your own package server, yes. It's not exactly the same thing. > Plone has set up their own mirroring of all the packages needed to set > up Plone, to avoid having problems with downed servers etc. That's not > a PyPI mirror, it's something else. It's their own independant package repository also because they want to have a "known good set of packages" like Turbogears or Zope has. And that's why at some point, package installers will probably need to merge serveral indexes that are not mirrors. This is what I've described in PEP 381 http://www.python.org/dev/peps/pep-0381 Notice that I've started this PEP right after I've set up plone.org's PyPI system at http://plone.org/products because in my mind, every framework should have its own package repository w.r.t the central PyPI. Tarek From regebro at gmail.com Thu Dec 24 14:56:26 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 14:56:26 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B335048.4060707@cryptelium.net> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> <4B335048.4060707@cryptelium.net> Message-ID: <319e029f0912240556g34cde9b5qce81faec5a6ea081@mail.gmail.com> On Thu, Dec 24, 2009 at 12:28, kiorky wrote: > Why forcing them ? > How about a cronjob or something like that which find packages without > distribution and get from there all distributions possible on their relative > homepages and mirror them on Pypi ? Somebody (including a bot) uploading a package is quite different from *requiring* uploads, IMO. But uploading to PyPI is for 99.999% of packages so easy that if it's not done there may be a reason why they don't want to. We may choose to ignore that, but we can only do it for licenses we know allow it, which again means that there may be packages listed on PyPI that are not hosted on PyPI. I personally don't understand why third-party services would need this. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Thu Dec 24 15:06:46 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 15:06:46 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <7CB35A3D-43E4-4FF4-ABC7-7D559B7A47F0@gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> <7CB35A3D-43E4-4FF4-ABC7-7D559B7A47F0@gmail.com> Message-ID: <319e029f0912240606o4d00ab96sa8ca6bb038318ce4@mail.gmail.com> On Thu, Dec 24, 2009 at 14:44, ssteinerX at gmail.com wrote: > Ok, so easy_install works for Twisted. ?Yay. So, your requirement is fulfilled. > I think that's exactly *not*: > > # super-duper-python-thing-just-like-cpan-only-better -i Twisted No, it's > # super-duper-python-thing-just-like-cpan-only-better -i Twisted and > # the-other-super-duper-python-thing-just-like-cpan-only-even-better -i Twisted > And, I picked Twisted 'cause it was already in the discussion. Yes, and this works pretty much for any other package as well. There are some that are tricky because they require C-libraries that are hard to install etc, but most packages can be installed like this. Complete applications, like Zope2, Turbogears etcs, can often not, but even the last version of Zope 2 can be easy_installed. > There are at least three ways to install it and I'm really concerned that what you just seemed to suggest was that the much-maligned-and-soon-to-be-deprecated `easy_install` is the `super-duper-python-thing-just-like-cpan-only-better` of today? That's a strange concern. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Thu Dec 24 15:06:53 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 15:06:53 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <87k4wc7eh5.fsf@benfinney.id.au> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <4B334273.1050707@v.loewis.de> <87k4wc7eh5.fsf@benfinney.id.au> Message-ID: <319e029f0912240606y7122ef82p4edb81b73c9bdc5b@mail.gmail.com> On Thu, Dec 24, 2009 at 13:17, Ben Finney wrote: >> Forcing people to do what you think they should do will not make them >> make more or better work. It will just make them do *less* work. > > That isn't a good argument. It's a *fantastic* argument. > By the same logic, PyPI should not reject > *any* upload, to avoid ?forcing? uploaders to do extra work. Uhhhh.... yes? Which of course is how it works? The comment confused me, I don't understand what you are trying to say. > No, PyPI needs to reject uploads on some criteria; those criteria will > necessarily involve work (whether manual or automated) on the part of > uploaders. This discussion is about what those criteria should be. No, it does not need to do that at all, and no, the discussion isn't about that in any way. The only discussion related to that has been about requiring the upload of the source. That's it. Nobody has to my knowledge suggested any other requirements. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From ssteinerx at gmail.com Thu Dec 24 15:46:58 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 24 Dec 2009 09:46:58 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912240606o4d00ab96sa8ca6bb038318ce4@mail.gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> <7CB35A3D-43E4-4FF4-ABC7-7D559B7A47F0@gmail.com> <319e029f0912240606o4d00ab96sa8ca6bb038318ce4@mail.gmail.com> Message-ID: On Dec 24, 2009, at 9:06 AM, Lennart Regebro wrote: >> I'm really concerned that what you just seemed to suggest was that the much-maligned-and-soon-to-be-deprecated `easy_install` is the `super-duper-python-thing-just-like-cpan-only-better` of today? > > That's a strange concern. I don't think it's strange to be concerned that one of the SDPTJLCOB options, and the one that you used as your example, is going to go away. If pip is going to be the SDPTJLCOB of the future, with an facade that substitutes for easy_install in the same way that Distribute does for setuptools then great, that's progress! S From regebro at gmail.com Thu Dec 24 16:16:59 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 16:16:59 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> <7CB35A3D-43E4-4FF4-ABC7-7D559B7A47F0@gmail.com> <319e029f0912240606o4d00ab96sa8ca6bb038318ce4@mail.gmail.com> Message-ID: <319e029f0912240716q2f528396w9498666f8536d41a@mail.gmail.com> On Thu, Dec 24, 2009 at 15:46, ssteinerX at gmail.com wrote: > I don't think it's strange to be concerned that one of the SDPTJLCOB options, and the one that you used as your example, is going to go away. Why? Pip does the same thing. They are commands. And it's not likely to go away anytime soon, what probably happens is that people will use pip more and more until pretty much noone uses easy_install. It completely and utterly fails me to see why this would be anything to worry about. > If pip is going to be the SDPTJLCOB of the future, with an facade that substitutes for easy_install in the same way that Distribute does for setuptools then great, that's progress! It'll have an easy_install facade if someone needs it badly enough to write one. But I can't imagine why they would. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From smueller at cpan.org Wed Dec 23 10:32:47 2009 From: smueller at cpan.org (smueller at cpan.org) Date: Wed, 23 Dec 2009 01:32:47 -0800 (PST) Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> Message-ID: Hi Lennart, thanks for the reply. Lennart Regebro gmail.com> writes: > On Tue, Dec 22, 2009 at 10:14, Steffen Mueller cpan.org> wrote: > > Let me add two nits here: > > It's Perl, not PERL. The name of the language is *not* an acronym. Some people > > are really picky about that. > > In a language where you can spell any functionality in millions of > ways, you are not allowed to spell the language however you want!? > Ok, P?rl. I mean Perl. IMO, the "There is more than one way to do it" mantra gets old very quick. It's generally appreciated as something along the lines of "There is more than one way to do it, but most of them are bad." You'd be surprised how many tools are converging into best practices in Perl. Cf. modules like Moose or DBIx::Class, etc. There's still at least one new, sucky configuration parser uploaded to CPAN every day, though :( Anyway. I appreciate the humour ;) > OK, so that would be docstrings and stuff then. It's true, if we had > something like that it would work as an incentive for making more such > documentation. I agree. That helps tremendously. > > You could get the same page via > > http://search.cpan.org/perldoc?PAR::Repository::Client. This is an example of > > how CPAN works on namespace and distribution level. After a new file is > > uploaded, its contained namespaces (packages/class names) are added to the > > index if they're not already there. > > How does it handle if two modules implement the same namespace? There are two ways to get a namespace (non-recursively!). Either you file a form with the PAUSE admins or you're simply the first to upload under the name. If someone does something outrageously stupid wrt. namespaces, he feels the wraith of the community. If that's not enough, the admins can act. But this is very, very, extremely rare. > > The index always contains a reference to > > the distribution that contains the newest *authorized* version of any > > namespace. > > Who authorizes it? The "first come first serve" principle. Or if there is a disagreement/ clash between authors, the PAUSE admins. > I have to say that I vastly prefer not to have any authorization and > allow anyone to release anything in any namespace. But then I am > getting fanatically anarchical in these issues. You can not organize > freedom. But you have to. What you're saying here means you virtually throw away the ability to do anything useful with namespace meta data. Think about it like this: If you install any module from CPAN (and only the valid ones end up in the index), you can use all of them in the same application. If module A and B could both implement Config::Parser, then you couldn't use both of them at the same time. Now, this may be a case where my lack of Python makes me look silly. In Perl, when I refer to namespaces, I'm actually thinking of hierarchical class structures, not C++ style namespaces which may in turn contain arbitrary classes. Effectively, though, the C++ namespaces + classes end up about the same as Perl's namespaces. Still. We allow for a lot of creative freedom. We just don't want a random newbie upload a shiny package "DB" which implements his idea of a database interface when it's really the name of the package that implements the Perl debugger. He can still upload. The file will be accessible in his CPAN directory. Users can install it from the CPAN shell as "install NEWBIEUSERID/DB-1.00.tar.gz", but not as "install DB" or "$ cpan DB". This is a safeguard against insanity and it's the thing that means that you can trust "cpan PAR" to always install the Perl Archive Toolkit that was released by Autrijus Tang, Roderich Schupp, or myself (we share co-maintenance). It's never going to be some random junk. And that you auto-register a namespace on upload is the guarantee. Best regards, Steffen PS: Let me comment on another post of yours quickly. No. In the Perl community, the name "CPAN" doesn't yield confusion. It's just a way to refer to the whole ecosystem including but not limited to: * the FTP/HTTP archive * PAUSE, the Perl Authors Upload Server and the indexes it generates * The various web frontends * The "toolchain", i.e. the set of tools that can create new distributions as well as build and install them. * The huge set of tests and modules that do the testing. * The CPAN.pm or CPANPLUS modules that provide the user interface (shell, etc) and do the dependency resolution, downloading, local unpacking, controlling of the build steps. * The "cpan" and "cpanp" CLIs to CPAN.pm and CPANPLUS respectively. Let me also refer to Ash Berlin's recent posts on a similar subject. PPS: All, could you please not turn this thread into a flame war? Civility and mutual respect go a long way towards getting shit the fuck done. Pun intended. :P From George.Dowding at netapp.com Wed Dec 23 16:35:53 2009 From: George.Dowding at netapp.com (Dowding, George) Date: Wed, 23 Dec 2009 10:35:53 -0500 Subject: [Distutils] Is it possible to configure __requires__ and load_entry_point for console_script? Message-ID: Hi, I have a setup() that uses a version control change number to set the version of the package. Part of the package includes some console_script directives. When the scripts are generated the __requires__ and load_entry_point will be set to the exact version of the package. I would like to have it set to something less restrictive. Is that possible? Example change = '12345' setup( name='my-project', version='my-project.0.1.dev-r%s' % version, entry_points={'console_scripts': ['my_script = my_module:main']}) Will generate my_script __requires__ = 'my-project==0.1.dev-r12345' import sys from pkg_resources import load_entry_point sys.exit( load_entry_point('my-project==0.1.dev-r12345', 'console_scripts', 'my_script')() ) I would like it to change the requirement such that the script will run with 'my-project==0.1.dev' regardless of the exact change number. -- Thanks, George A. Dowding -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssteinerx at gmail.com Thu Dec 24 17:48:14 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 24 Dec 2009 11:48:14 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912240716q2f528396w9498666f8536d41a@mail.gmail.com> References: <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> <7CB35A3D-43E4-4FF4-ABC7-7D559B7A47F0@gmail.com> <319e029f0912240606o4d00ab96sa8ca6bb038318ce4@mail.gmail.com> <319e029f0912240716q2f528396w9498666f8536d41a@mail.gmail.com> Message-ID: On Dec 24, 2009, at 10:16 AM, Lennart Regebro wrote: > On Thu, Dec 24, 2009 at 15:46, ssteinerX at gmail.com wrote: >> I don't think it's strange to be concerned that one of the SDPTJLCOB options, and the one that you used as your example, is going to go away. > > Why? Pip does the same thing. They are commands. And it's not likely > to go away anytime soon, what probably happens is that people will use > pip more and more until pretty much noone uses easy_install. It > completely and utterly fails me to see why this would be anything to > worry about. I guess what I mean is that I'd like to make sure that while moving to pip, that easy_install, as a command name, not as an implementation, gets brought along in the same way that Distribute has brought along setuptools compatibility. IOW in such a way that the improvements in the new code are available without breaking any existing configurations/scripts/workflows etc. >> If pip is going to be the SDPTJLCOB of the future, with an facade that substitutes for easy_install in the same way that Distribute does for setuptools then great, that's progress! > > It'll have an easy_install facade if someone needs it badly enough to write one. But I can't imagine why they would. I imagine it would be useful to have a facade because easy_install is familiar and ubiquitous and shouldn't be left rotting in place as pip takes over the world and becomes the SDPTJLCOB any more than Distribute left the rest of setuptools exposed to cause problems once Distribute was installed. S From kiorky at cryptelium.net Thu Dec 24 22:20:37 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 24 Dec 2009 22:20:37 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912240556g34cde9b5qce81faec5a6ea081@mail.gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> <4B335048.4060707@cryptelium.net> <319e029f0912240556g34cde9b5qce81faec5a6ea081@mail.gmail.com> Message-ID: <4B33DB25.7010608@cryptelium.net> Lennart Regebro a ?crit : > On Thu, Dec 24, 2009 at 12:28, kiorky wrote: >> Why forcing them ? >> How about a cronjob or something like that which find packages without >> distribution and get from there all distributions possible on their relative >> homepages and mirror them on Pypi ? > > Somebody (including a bot) uploading a package is quite different from > *requiring* uploads, IMO. > > But uploading to PyPI is for 99.999% of packages so easy that if it's > not done there may be a reason why they don't want to. Unless they don't know how to do. > We may choose to ignore that, but we can only do it for licenses we know allow it, 99% of pypi packages. > which again means that there may be packages listed on PyPI that are > not hosted on PyPI. > > I personally don't understand why third-party services would need this. > That's quite out of scope which what i wanted to spot but to answear: The only problem is with licenses which constrain redistribution. We can have logic around trove classifiers to filter them out. If the authors miss the trove, that's their fault. What i would like to see is just to have most of the distributions on Pypi because by experience, pypi get less offline that thirdparty mirrors, and i can set mirrors of pypi easyly. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From ben+python at benfinney.id.au Thu Dec 24 22:45:04 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 25 Dec 2009 08:45:04 +1100 Subject: [Distutils] Python people want CPAN and how the latter came about References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <4B334273.1050707@v.loewis.de> <87k4wc7eh5.fsf@benfinney.id.au> <4B336D08.8060308@v.loewis.de> Message-ID: <87fx706o7z.fsf@benfinney.id.au> "Martin v. L?wis" writes: > Ben Finney wrote: > > That isn't a good argument. By the same logic, PyPI should not > > reject *any* upload, to avoid ?forcing? uploaders to do extra work. > > PyPI's rejection of certain uploads is primarily to prevent spam from > being uploaded. Am I wrong, then, in thinking that PyPI will reject an upload with malformed metadata? To my understanding, this discussion is about arguing whether an upload that is missing the package should be rejected by PyPI. -- \ ?Absurdity, n. A statement or belief manifestly inconsistent | `\ with one's own opinion.? ?Ambrose Bierce, _The Devil's | _o__) Dictionary_, 1906 | Ben Finney From regebro at gmail.com Thu Dec 24 23:50:12 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 23:50:12 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> <7CB35A3D-43E4-4FF4-ABC7-7D559B7A47F0@gmail.com> <319e029f0912240606o4d00ab96sa8ca6bb038318ce4@mail.gmail.com> <319e029f0912240716q2f528396w9498666f8536d41a@mail.gmail.com> Message-ID: <319e029f0912241450k7518597cl1974eae41718fcb9@mail.gmail.com> On Thu, Dec 24, 2009 at 17:48, ssteinerX at gmail.com wrote: > I guess what I mean is that I'd like to make sure that while moving to pip, that easy_install, as a command name, not as an implementation, gets brought along in the same way that Distribute has brought along setuptools compatibility. ?IOW in such a way that the improvements in the new code are available without breaking any existing configurations/scripts/workflows etc. easy_install is a command, basically a wrapper around setuptools install functionality. I doubt many scripts would use it in any more complex way than calling it, in which case moving to pip is a matter of replacing the command. > I imagine it would be useful to have a facade because easy_install is familiar and ubiquitous and shouldn't be left rotting in place as pip takes over the world and becomes the SDPTJLCOB any more than Distribute left the rest of setuptools exposed to cause problems once Distribute was installed. I don't even understand that sentence, let alone what you are trying to say. Distribute is a setuptools fork. Pip is not an easy_install fork. I don't really see the parallell. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Thu Dec 24 23:52:21 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 23:52:21 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B33DB25.7010608@cryptelium.net> References: <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> <4B335048.4060707@cryptelium.net> <319e029f0912240556g34cde9b5qce81faec5a6ea081@mail.gmail.com> <4B33DB25.7010608@cryptelium.net> Message-ID: <319e029f0912241452r37678391h9a28c60bcb9b70f7@mail.gmail.com> On Thu, Dec 24, 2009 at 22:20, kiorky wrote: >> But uploading to PyPI is for 99.999% of packages so easy that if it's >> not done there may be a reason why they don't want to. > > Unless they don't know how to do. It's well documented and very easy. > What i would like to see is just to have most of the distributions on Pypi Me too. I just said that I don't think we should require it. Help people to make it easier in various ways I'm all for. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Thu Dec 24 23:56:32 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 24 Dec 2009 23:56:32 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <87fx706o7z.fsf@benfinney.id.au> References: <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <4B334273.1050707@v.loewis.de> <87k4wc7eh5.fsf@benfinney.id.au> <4B336D08.8060308@v.loewis.de> <87fx706o7z.fsf@benfinney.id.au> Message-ID: <319e029f0912241456u6a3caf3fm2905b000318eb264@mail.gmail.com> On Thu, Dec 24, 2009 at 22:45, Ben Finney wrote: > Am I wrong, then, in thinking that PyPI will reject an upload with > malformed metadata? Well, define "malformed". If the data format is impossible to interpret, it obviously have to fail. That's not a reject in any big sense. There are also required metadata, namely package name and version. > To my understanding, this discussion is about arguing whether an upload > that is missing the package should be rejected by PyPI. Well, it's about if a package that is missing any upload should be "rejected" (or shown, rather, since registering packages and uploading files are separate in PyPI). -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Fri Dec 25 00:14:59 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 00:14:59 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> Message-ID: <319e029f0912241514y1e818590xf8cb9d52dd9ed477@mail.gmail.com> On Wed, Dec 23, 2009 at 10:32, smueller at cpan.org wrote: >> I have to say that I vastly prefer not to have any authorization and >> allow anyone to release anything in any namespace. But then I am >> getting fanatically anarchical in these issues. You can not organize >> freedom. > > But you have to. No, I really mean it whan I say you can't. And you never *have* to do the impossible, and trying just leads to problems. I realize this is a matter of attitude, but if the sentence "I want CPAN" means "I want restrictions and controls and people preventing others from uploading stuff", then they are misguided. > What you're saying here means you virtually throw > away the ability to do anything useful with namespace meta data. *shrugs* Namespaces has no metadata in Python. They are just namespaces, no more, no less. The names of your *packages* is protected on PyPI. But several people can use the same *namespace*. Ie, nobody can upload a "Twisted" package, except those who have the permission. But people can upload a "zope.my.own.package", even though the namespace "zope" is already used by other packages. > Think about it like this: If you install any module from CPAN (and > only the valid ones end up in the index), you can use all of them in > the same application. If module A and B could both implement > Config::Parser, then you couldn't use both of them at the same time. This would be true for Python too. But Python doesn't try to pretend that all the packages that exist are some sort of standard library, and therefore don't try to put them all into one sort of hierarchical organized namespace. And to be honest I don't see the point of doing that. > Still. We allow for a lot of creative freedom. We just don't want a > random newbie upload a shiny package "DB" which implements his idea of > a database interface when it's really the name of the package that > implements the Perl debugger. He can still upload. The file will be > accessible in his CPAN directory. Users can install it from the CPAN > shell as "install NEWBIEUSERID/DB-1.00.tar.gz", but not as "install > DB" or "$ cpan DB". I see. But IMO Perl then there starts out with trying to organize freedom from the start, and that then leads to the above problem that newbies can come up and mess up this so called organized freedom, which means you need to restrict it even more by having people control and restrict the namespaces, etc, etc. You end up having to have more organisation to fix the problems your organisation caused. This is, without trying to be rude or anything", the fate of all bureaucracies. > This is a safeguard against insanity and it's the thing that means > that you can trust "cpan PAR" to always install the Perl Archive > Toolkit that was released by Autrijus Tang, Roderich Schupp, or myself > (we share co-maintenance). It's never going to be some random junk. > And that you auto-register a namespace on upload is the guarantee. And obviously on PyPI, it's first come first serve as well. But nobody would call a db package "db" if one already exists. Why would they do that? What's the point? Why would I make a completely new package called "Twisted" for example? There already is one. It's just a mindset that is completely incomprehensible to me. I expect what I would call creative freedom, you would call total anarchy. > PS: Let me comment on another post of yours quickly. No. In the Perl > community, the name "CPAN" doesn't yield confusion. It's just a way to > refer to the whole ecosystem OK, that's not how it sounded in your first post, thanks for clarifying. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From ssteinerx at gmail.com Fri Dec 25 01:16:54 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 24 Dec 2009 19:16:54 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912241450k7518597cl1974eae41718fcb9@mail.gmail.com> References: <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <71DC1779-7A18-4E31-8CEC-C2071A6B2CCD@gmail.com> <319e029f0912232302k181d60bap92b629eedd987a32@mail.gmail.com> <7CB35A3D-43E4-4FF4-ABC7-7D559B7A47F0@gmail.com> <319e029f0912240606o4d00ab96sa8ca6bb038318ce4@mail.gmail.com> <319e029f0912240716q2f528396w9498666f8536d41a@mail.gmail.com> <319e029f0912241450k7518597cl1974eae41718fcb9@mail.gmail.com> Message-ID: <7DBD7332-F374-4330-8EF4-0CCABA22EBF4@gmail.com> On Dec 24, 2009, at 5:50 PM, Lennart Regebro wrote: > easy_install is a command, basically a wrapper around setuptools > install functionality. I doubt many scripts would use it in any more > complex way than calling it, in which case moving to pip is a matter > of replacing the command. Yes, I'm saying that it would be smart to implement a replacement easy_install with pip doing the work in the background instead of setuptools so that no changes are necessary. >> I imagine it would be useful to have a facade because easy_install is familiar and ubiquitous and shouldn't be left rotting in place as pip takes over the world and becomes the SDPTJLCOB any more than Distribute left the rest of setuptools exposed to cause problems once Distribute was installed. > > I don't even understand that sentence, let alone what you are trying > to say. Distribute is a setuptools fork. Distribute is a setuptools fork and is intended to replace it in the Python ecosystem. In order to accomplish this, it transparently replaces setuptools. > Pip is not an easy_install fork. I don't really see the parallell. The parallel is that as Distribute transparently replaces setuptools, the new easy_install, with pip doing the behind-the-scenes work, should transparently replace the existing easy_install. # easy_install Twisted should still work just as it does now, it'll just do a better job of it with the new machinery running in the background. S From sridharr at activestate.com Fri Dec 25 05:39:37 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 24 Dec 2009 20:39:37 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B33276E.2070900@v.loewis.de> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> Message-ID: <4B344209.3090209@activestate.com> On 12/24/2009 12:33 AM, "Martin v. L?wis" wrote: >> 1/ Missing packages (eg: Twisted is not there); which is why >> easy_install/pip had to resolve to scrapping project webpages for >> guessing download links. In CPAN, almost all module authors upload their >> sources via PAUSE. > > How do you propose to change that? Bt requiring authors to upload sdists + metadata now onwards. 'sdist upload' would upload the sdist to /packages/source and also have PyPI generate the metadata from the uploaded sdist. Eg: /packages/source/f/foo-0.1.tar.gz /packages/source/f/foo-0.1.tar.gz.PKG-INFO /packages/source/f/foo-0.1.tar.gz.requires.txt (optional) If the author prefers to use the web browser to upload, then their sdist must contain setup.py and PKG-INFO (w/ at least 'name' and 'version'). I would leave the existing setup as it is .. so easy_install/pip would continue to install packages like Twisted, ClientCookie that, at the moment, do not have their sdists uploaded in PyPI. [Martin] >>> I think it should be the choice of the package authors whether they >>> upload their software to the central repository, or to their own home >>> page. >> >> [Ben] >> Why do you think that should continue? Some of the costs of that >> inconsistency have already been described in this thread. What are the >> benefits to PyPI users of this inconsistency, and are we sure that the >> benefits outweigh the costs? > > [Martin] > The benefits are not to the package users, clearly.Instead, they are > to the package authors, which don't have to change their release > processes (as also described in this thread). Is it because of this benefit to package authors that we are withholding the implementation of a simple archive that would: 1) simplify the tools to no rely on adhoc web scrapping, 2) reduce the downtime for users by rsync/ftp mirroring, 3) have package sources mirrored so project owners do not have to worry about downtime of their servers. 4) enable proliferation of third-party tools like CPAN? >> 2/ No metadata: When only source tarballs are stored >> [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to >> a) get the source for latest version, > > Extract a version number from each file name, and sort the versions, > then use the largest (which is 0.9.7 at the moment). > >> b) get the source for a particular version? > > Put the version number into the file name, and access the resulting > file. This assumes that source tarballs are named in a particular format, such as: ${name}-${version}.tar.gz .. which need not always be the case (I've come across packages whose source distribution is simply named "latest"). This is why we rely on PKG-INFO to retrieve the version. The reason for asking the two questions above, as pointed out to Lennart in other email, is this: """Perhaps if I were to rephrase the question, it would be clear this time: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for the latest version (when the /P/Pylons contains multiple versions -- in other words, how do I find the later version in first place?), b) get the source for a particular version (**without** having to construct the filename, or do a adhoc matching with filenames to guess that Pylons-1.2.3.tar.gz corresponds to version 1.2.3)? If the answer is to do a HTTP GET first, then please see the next response. """ [emphasis added] My next response was: """As the CPAN .meta example was given in the context of having a simple directory structure that can be mirrored using existing tools like rsync, what I was pointing out is the lack of such an implementation, not the functionality itself (which, as you have shown, is currently supported by doing a HTTP GET that would return a XML content -- not something that is rsync-friendly). """ To explain: it is all about making the PyPI data (sdist + metadata) mirror-friendly / rsync-friendly. >> The former is more of a community issue. Often Python package authors >> are not using `sdist upload` (whereas this seems to be the convention in >> the Perl world). > > My guess is that this is enforced by the tools. If they don't upload > to PAUSE, CPAN.pm won't be able to download it. > > Now, you are free to build a tool that enforces the same restriction. > I would doubt that people would use it, since it couldn't install > many packages. My original intention is to have a simple archive that can be mirroed using rsync. >> What this means is that PyPI has to serve the purpose of being a central >> package repository (like CPAN) by a) disallowing mere listings (without >> sources) and requiring sources to be stored in the server, b) storing >> the metadata along with the sources (so anyone processing it wouldn't >> have to extract the source and rely on a PKG-INFO file - which may or >> may not exist). > > If you want to retrieve the metadata for a specific version without > XML-RPC, you can access > > http://pypi.python.org/pypi?:action=doap&name=Pylons&version=0.9.7 As pointed above, the purpose is not to do away with XmlRpc as such, but to have a simple archive that can be mirrored in entirety using existing tools like rsync. To facilitate this, one should be able to retrieve the metadata from the archive itself (filesystem) instead of having to do HTTP requests (via plain GET or XmlRpc). -srid From sridharr at activestate.com Fri Dec 25 05:48:16 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 24 Dec 2009 20:48:16 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912232242uc02eef2i8c261783e1622e8c@mail.gmail.com> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B32997A.3070407@activestate.com> <319e029f0912232242uc02eef2i8c261783e1622e8c@mail.gmail.com> Message-ID: <4B344410.5010308@activestate.com> On 12/23/2009 10:42 PM, Lennart Regebro wrote: > On Wed, Dec 23, 2009 at 23:28, Sridhar Ratnakumar > wrote: >> I suggested PyPI to disallow mere project listings (without sources) and >> require sources to be stored in the server. One way to achieve this is >> requiring package authors to use the `sdist upload` toolchain > > Which only means the packages who now is not uploaded wouldn't even be > listed on PyPI, which is not an improvement. We can do this only for the new projects/uploads. Existing data can be left as it is for backwards compatibility. Here's my updated proposal: [Sridhar's proposal] >> How do you propose to change that? > > By requiring authors to upload sdists + metadata now onwards. > > 'sdist upload' would upload the sdist to /packages/source and also have PyPI generate the metadata from the uploaded sdist. Eg: > > /packages/source/f/foo-0.1.tar.gz > /packages/source/f/foo-0.1.tar.gz.PKG-INFO > /packages/source/f/foo-0.1.tar.gz.requires.txt (optional) > > If the author prefers to use the web browser to upload, then their sdist must contain setup.py and PKG-INFO (w/ at least 'name' and 'version'). > > I would leave the existing setup as it is .. so easy_install/pip would continue to install packages like Twisted, ClientCookie that, at the moment, do not have their sdists uploaded in PyPI. ... >> While the specific case mentioned above (metadata for a specific or the >> latest version of a package) uses HTTP GET and XML, generally speaking .. to >> get a) the list of recently releases, b) list of all versions of a package, >> one has to use the XmlRpc API methods `changelog` and `package_releases` >> respectively. > > Well, maybe pure http versions of those would help, Nope, it matters not whether the metadata can be retrived via a simple HTTP GET or XmlRpc. > but on the other > hand, if you automate it, why not use xml-rpc? Because my intention is to have a simple mirror archive (files, directories) that can be mirrored using tools like rsync. >> As often as the mirror sites would update their content (i.e., one or more >> times a day). > > I meant that most of the third-party apps would only need the > metadata, or? I might be wrong, I haven't written any yet. :-) The > automated documentation that was discussed would only need the source > packages. Metadata is definitely needed. Otherwise, I'd have to extract the tarball of each and every release of a pacticular package, in order to even find their version number (it is unreliable to parse the filename to get version number). As for the sdists, the following tools would need it: testing service, quality ratings, thirdparty package managers (enstaller, PyPM) .. and not to mention the various mirror sites. -srid From tseaver at palladion.com Fri Dec 25 05:54:34 2009 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 24 Dec 2009 23:54:34 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <87fx706o7z.fsf@benfinney.id.au> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <4B334273.1050707@v.loewis.de> <87k4wc7eh5.fsf@benfinney.id.au> <4B336D08.8060308@v.loewis.de> <87fx706o7z.fsf@benfinney.id.au> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ben Finney wrote: > "Martin v. L?wis" writes: > >> Ben Finney wrote: >>> That isn't a good argument. By the same logic, PyPI should not >>> reject *any* upload, to avoid ?forcing? uploaders to do extra work. >> PyPI's rejection of certain uploads is primarily to prevent spam from >> being uploaded. > > Am I wrong, then, in thinking that PyPI will reject an upload with > malformed metadata? > > To my understanding, this discussion is about arguing whether an upload > that is missing the package should be rejected by PyPI. In the language of distutils, recording the metadata about a release is "registration"; registration and upload happen in separate transactions for relatively good reasone: - - There is not a one-to-one relationsihp between metadata sets (registrations) and distributions (e.g., source dist + windows MSI). - - PiPI only has to parse the PKG-iNFO file, and doesn't need to unpack a distribution to look for it. - - Package authors can choose to keep the actual distribution files elsewhere, e.g., to allow for payment, etc.: one might debate the desirability of such a use case for the community at large, but it is certainly part of the historical use of the cheeseshop. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks0RYUACgkQ+gerLs4ltQ52GQCdFhOUpq9c4hcN7vHiFGaOfqm0 ufUAoMc5FtMKyd5GZI2WySsoUgcgbzj9 =dib1 -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Dec 25 05:58:27 2009 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 24 Dec 2009 23:58:27 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B33DB25.7010608@cryptelium.net> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> <4B335048.4060707@cryptelium.net> <319e029f0912240556g34cde9b5qce81faec5a6ea081@mail.gmail.com> <4B33DB25.7010608@cryptelium.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 kiorky wrote: > > Lennart Regebro a ?crit : >> On Thu, Dec 24, 2009 at 12:28, kiorky wrote: >>> Why forcing them ? >>> How about a cronjob or something like that which find packages without >>> distribution and get from there all distributions possible on their relative >>> homepages and mirror them on Pypi ? >> Somebody (including a bot) uploading a package is quite different from >> *requiring* uploads, IMO. >> >> But uploading to PyPI is for 99.999% of packages so easy that if it's >> not done there may be a reason why they don't want to. > > Unless they don't know how to do. > >> We may choose to ignore that, but we can only do it for licenses we know allow it, > 99% of pypi packages. > >> which again means that there may be packages listed on PyPI that are >> not hosted on PyPI. >> >> I personally don't understand why third-party services would need this. >> > > That's quite out of scope which what i wanted to spot but to answear: > > The only problem is with licenses which constrain redistribution. > We can have logic around trove classifiers to filter them out. > If the authors miss the trove, that's their fault. > > What i would like to see is just to have most of the distributions on Pypi > because by experience, pypi get less offline that thirdparty mirrors, and i can > set mirrors of pypi easyly. I would say that having a package author *not* upload the distributions is their right, but I would likely avoid using such a package, just on that basis. Note that I build per-project mirrors of the pacakges I use anyway, in part not to depend on either PyPI or other download sources for supporting apps in production: I just prefer to use only freely-distributable software. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks0RnIACgkQ+gerLs4ltQ61FACgnE0jgLx5jaHTQzMofH+VyT5I YBMAn079ghQ2lLcOwUraISeew81Pe9jt =gUew -----END PGP SIGNATURE----- From sridharr at activestate.com Fri Dec 25 06:09:04 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 24 Dec 2009 21:09:04 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B3349B5.3000804@v.loewis.de> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <20091224104658.GA12434@rivendell.internal> <4B3349B5.3000804@v.loewis.de> Message-ID: <4B3448F0.4090808@activestate.com> On 12/24/2009 3:00 AM, "Martin v. L?wis" wrote: >> Some reasons to have PyPI host packages have already been mentioned in >> > this thread: it makes mirroring easier, and it makes it easier for >> > individuals to build new services (web sites primarily) that present new >> > interfaces to the Python package collection. Mirroring for its own sake >> > is some use, but being able to grab the entire Python package repository >> > easily from a single source is valuable for the second goal, that of >> > furnishing the foundation ("shoulders of giants" and all that) for those >> > with vision (and round tuits) to take the next step. > That is fairly easily possible today, even without everybody uploading > all files. It isn't easy*per se*, but needs a lot of code. However, > this code has already been written, and using it is fairly easy. > >> > If I wanted to host a site that (e.g.) indexed Python modules from PyPI >> > by module (not package) name, and extracted and provided the >> > documentation in HTML format, from what I've been reading I'd have to >> > build a scraper or XMLRPC tool to walk PyPI, and then for each package, >> > download it from another site (that may not have the uptime or >> > scalability of PyPI), a nontrivial burden on aspiring visionaries that >> > just want to build an addition and then go have a beer and discuss >> > further improvements. > Not at all. You would just use one of the ten or so packages that > already do precisely that, and use it. > >> > (As a point of practical interest, what_would_ be the most efficient >> > way to download the entire set of Python modules listed on PyPI? A >> > search comes up with z3c.pypimirror, >> > http://pypi.python.org/pypi/z3c.pypimirror; is this the standard tool?) > There are a number of other mirroring tools, such as EggBasket and > collective.eggproxy. For mirroring the whole index, pypimirror is > probably the best starting point. For starters, this is how z3c.pypimirror works: 1) Initial fetch: Traverse http://pypi.python.org/simple/AOPython and for each package's index.html, for each link in it, if the link is a) an actual sdist tarball, download it, b) an external link (project homepage), go scrap the homepage to find any download links (.tar.gz, .zip) and download it. 2) [run this every day] Update for last 24hrs: Use XmlRpc `changelog` method that returns recently released packages in last 24 hrs; and redo the above operation for these updated packages. It is unreliable [bugs.launchpad.net/pypi-mirror/+bug/386143] and lacks the pre-extracted metadata. I wouldn't call it a mirror tool, for it is not an exact copy of PyPI data[1]. I doubt that proliferation of mirror sites / thirdparty tools can happen with anything but simple rsync/ftp based archives. -srid *** [1] "In computing, a mirror is an exact copy of a data set. On the Internet, a mirror site is an exact copy of another Internet site. ..." From tseaver at palladion.com Fri Dec 25 06:12:55 2009 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 25 Dec 2009 00:12:55 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <94bdd2610912240244q385fc4ebg7d86aeb19610afbe@mail.gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> <4B333D3B.7010403@cryptelium.net> <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> <4B334068.2060900@cryptelium.net> <94bdd2610912240244q385fc4ebg7d86aeb19610afbe@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tarek Ziad? wrote: > On Thu, Dec 24, 2009 at 11:20 AM, kiorky wrote: > [..] >>> That said, some people have expressed the desire to be able to >>> interact with PyPI through SSH so they could drop the basic >>> authentication and use their keys when registering/uploading. >>> But that was not related to the size of files. >>> >> Both i think, it's easier to use the transfer programs that we are used to use >> to transfer things. >> And ssh is damn good for that, easy to setup, secure and so on. >> >> But it's only butter, in fact i am just happy with sdist upload. > > Although SSH is quite a heavy development on PyPI side, it means we > would have to implement > an SSH server. (like Zope did I think for their development server, > using Paramiko IIRC) cvs.zope.org / svn.zope.org (same machine) run a stock sshd: they use the "command=" prefix on users' pubkeys to limit what that key can be used to do (only SVN / CVS operations for any non-admin users). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks0SdcACgkQ+gerLs4ltQ795gCg2FDY353KLgJTXGgqz0CHQ1rE 2/UAoI7kUnEGxF2omBqh58JKSsFKEGVr =yRkp -----END PGP SIGNATURE----- From regebro at gmail.com Fri Dec 25 07:27:31 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 07:27:31 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B344209.3090209@activestate.com> References: <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> Message-ID: <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> On Fri, Dec 25, 2009 at 05:39, Sridhar Ratnakumar wrote: > Is it because of this benefit to package authors that we are withholding the > implementation of a simple archive that would: 1) simplify the tools to no > rely on adhoc web scrapping There are better ways to do that. > 2) reduce the downtime for users by rsync/ftp mirroring This is true, but the idea to upload them by robots is preferable in my opinion. Again it's a difference between trying to force other people to behave to your expectations vs trying to make it easier for others to behave to your expectations. > 3) have package sources mirrored so project owners do not have to > worry about downtime of their servers. That's *their* problem. If they don't want to upload, then they don't want to upload. > 4) enable proliferation of third-party tools like CPAN? That won't help. On Fri, Dec 25, 2009 at 05:48, Sridhar Ratnakumar wrote: > On 12/23/2009 10:42 PM, Lennart Regebro wrote: >> >> On Wed, Dec 23, 2009 at 23:28, Sridhar Ratnakumar >> wrote: >>> >>> I suggested PyPI to disallow mere project listings (without sources) and >>> require sources to be stored in the server. One way to achieve this is >>> requiring package authors to use the `sdist upload` toolchain >> >> Which only means the packages who now is not uploaded wouldn't even be >> listed on PyPI, which is not an improvement. > > We can do this only for the new projects/uploads. Existing data can be left > as it is for backwards compatibility. Which only means the packages who in the future will not be uploaded will not even be listed on PyPI, which is not an improvement. > Nope, it matters not whether the metadata can be retrived via a simple HTTP > GET or XmlRpc. OK. Then you have two proposals: 1. Require uploading, which is a bad idea and 2. Making it easier to mirror the metadata, which seems reasonable, assuming it's currently hard. :) > Metadata is definitely needed. Otherwise, I'd have to extract the tarball of > each and every release of a pacticular package, in order to even find their > version number (it is unreliable to parse the filename to get version > number). Yes, but it's not particularly unreliable to compare the filename to see if it had been handled before. You don't even need to parse the version number for most services that work on the tarballs. > As for the sdists, the following tools would need it: testing service, > quality ratings, thirdparty package managers (enstaller, PyPM) .. and not to > mention the various mirror sites. Yes, but since thay have the source package, and will have to unpack it and build the packages anyway, they also have the metadata. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From cournape at gmail.com Fri Dec 25 08:22:59 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 25 Dec 2009 16:22:59 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> References: <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> Message-ID: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> On Fri, Dec 25, 2009 at 3:27 PM, Lennart Regebro wrote: > This is true, but the idea to upload them by robots is preferable in > my opinion. Again it's a difference between trying to force other > people to behave to your expectations vs trying to make it easier for > others to behave to your expectations. Is there any hard data to back up the idea that making some things mandatory when registering/uploading to Pypi is detrimental to Pypi's goal, or is it just opinion ? It is very difficult for me to understand the rationale for this. This is specific to Pypi AFAIK (compared to CRAN, hackage, etc...), and the advantages of making it mandatory are obvious and hard facts (easy to mirror, easy to do basic consistency checks, easy to interoperate), whereas the downsides are far from obvious. cheers, David From sridharr at activestate.com Fri Dec 25 09:00:41 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Fri, 25 Dec 2009 00:00:41 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> References: <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> Message-ID: <4B347129.1060106@activestate.com> Greetings Lennart, On 12/24/2009 10:27 PM, Lennart Regebro wrote: > On Fri, Dec 25, 2009 at 05:39, Sridhar Ratnakumar > wrote: >> Is it because of this benefit to package authors that we are withholding the >> implementation of a simple archive that would: 1) simplify the tools to no >> rely on adhoc web scrapping > > There are better ways to do that. May I ask, what would they be? >> 2) reduce the downtime for users by rsync/ftp mirroring > > This is true, but the idea to upload them by robots is preferable in > my opinion. Again it's a difference between trying to force other > people to behave to your expectations vs trying to make it easier for > others to behave to your expectations. > >> 3) have package sources mirrored so project owners do not have to >> worry about downtime of their servers. > > That's *their* problem. If they don't want to upload, then they don't > want to upload. As the original proposal is to retain the existing behavior for already registered/uploaded package releases (such as Twisted) so existing systems will continue to work, but implement the suggested upload rules only for new requests (creation/register)- so as to gradually improve the quality of PyPI like that of other packaging systems - by encouraging authors to generate a reasonably good sdist (setup.py + PKG-INFO) and uploading them .. and consequently enabling the move towards a static archive that can easily be mirrored, I fail to see just what good is achieved by retaining the status quo. If I want to use a web service, I obviously have to adhere to their rules and policies. Nobody is forcing me to do so. I assume in good faith that package authors will be happy to adapt to the new system .. for the benefit of everyone. I will be happy to be proven otherwise. (Speculations are useless; how about we actually ask the package authors themselves?) >> 4) enable proliferation of third-party tools like CPAN? > > That won't help. Why not? Do you conceive of any reason apart from CPAN-like archives that would help in proliferation of mirror sites and third-party sites? I ask because I personally went through significant hurdles to setup a daily PyPI mirror-like area. I just don't see how someone merely interested in writing a third-party service, or setup a mirror of PyPI would be *most likely inclined* to face similar hurdles before giving up. Because I went through these hurdles, I was able to appreciate CPAN's design while reading about it [cpan.org/misc/ZCAN.html]. >> Nope, it matters not whether the metadata can be retrived via a simple HTTP >> GET or XmlRpc. > > OK. Then you have two proposals: 1. Require uploading, which is a bad > idea and 2. Making it easier to mirror the metadata, which seems > reasonable, assuming it's currently hard. :) Here's one idea (example only): $ tar zxf foo-0.1.tar.gz $ cp foo-0.1/PKG-INFO foo-0.1.tar.gz.PKG-INFO >> Metadata is definitely needed. Otherwise, I'd have to extract the tarball of >> each and every release of a pacticular package, in order to even find their >> version number (it is unreliable to parse the filename to get version >> number). > > Yes, but it's not particularly unreliable to compare the filename to > see if it had been handled before. You don't even need to parse the > version number for most services that work on the tarballs. It is indeed unreliable to rely on filenames to get package versions (unless that sdist is generated by the `setup.py sdist` command). As I've mentioned elsewhere, some packages have weird filenames (eg: "latest.zip", "foo.py"); some others have '.dev' suffix in the filenames while setup.py:version (hence PKG-INFO) will not have the '.dev' prefix. And several other issues that I cannot recall right now. I am not speculating as I've actually experimented with the PyPI index, mirroring it .. handling the metadata in packages, and building it. >> As for the sdists, the following tools would need it: testing service, >> quality ratings, thirdparty package managers (enstaller, PyPM) .. and not to >> mention the various mirror sites. > > Yes, but since thay have the source package, and will have to unpack > it and build the packages anyway, they also have the metadata. It is not that simple. PyPM backend, for instance, is not monolithic as in doing only a sequential build of packages. It first loads the dependency graph (for which metadata - PKG-INFO/requires.txt - is required) from our internal mirror over the network. It is expensive to go extract each and every tarball .. from each build machine. After loading the dependency graph, and then comparing it with existing repository .. every day, new builds happen. Certain packages even lack metadata (eg: no PKG-INFO in Twisted's sdist) in their source distributions .. which is another issue altogether. Further, I can imagine search.cpan.org (which is not hosted by cpan.org folks) using only the metadata without touching the source distributions. -srid From regebro at gmail.com Fri Dec 25 09:52:35 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 09:52:35 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> Message-ID: <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> On Fri, Dec 25, 2009 at 08:22, David Cournapeau wrote: > Is there any hard data to back up the idea that making some things > mandatory when registering/uploading to Pypi is detrimental to Pypi's > goal, or is it just opinion ? It's just how humans work. Hard data comes from 200 years of economic and social sciences. > It is very difficult for me to understand the rationale for this. This > is specific to Pypi AFAIK Not at all. > (compared to CRAN, hackage, etc...), and the > advantages of making it mandatory are obvious and hard facts (easy to > mirror It does not make it more easy to mirror. When you mirror PyPI's file storage, you obviously do not mirror what is not there. > easy to do basic consistency checks, easy to interoperate), I don't see how that helps either. > whereas the downsides are far from obvious. The downside is *extremely* obvious: You make it harder to register packages on PyPI. That means less packages will be registered. As it stands today, the requirement that you would have to upload the packages to PyPI as well as register them means, for example, that neither Twisted nor PIL would not be listed on PyPI. Two very popular and well respected packages. That *is* an obvious downside. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cournape at gmail.com Fri Dec 25 10:06:50 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 25 Dec 2009 18:06:50 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> Message-ID: <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> On Fri, Dec 25, 2009 at 5:52 PM, Lennart Regebro wrote: >> It is very difficult for me to understand the rationale for this. This >> is specific to Pypi AFAIK > > Not at all. Which other system similar to Pypi do you know which works like Pypi in that regard ? Neither CRAN, hackage, opensuse build service, rpm/deb repositories work like this to cite the ones I am somehow familiar with. It seems that CPAN does not either if I understand from the discussion. David From regebro at gmail.com Fri Dec 25 10:09:29 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 10:09:29 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B347129.1060106@activestate.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <4B347129.1060106@activestate.com> Message-ID: <319e029f0912250109r63f44df2i679e5e06dac5e875@mail.gmail.com> On Fri, Dec 25, 2009 at 09:00, Sridhar Ratnakumar wrote: > Greetings Lennart, > > On 12/24/2009 10:27 PM, Lennart Regebro wrote: >> >> On Fri, Dec 25, 2009 at 05:39, Sridhar Ratnakumar >> ?wrote: >>> >>> Is it because of this benefit to package authors that we are withholding >>> the implementation of a simple archive that would: 1) simplify the tools >>> to no rely on adhoc web scrapping >> >> There are better ways to do that. > > May I ask, what would they be? Have links in the metadata to the file locations. That means you don't have to scrape the websites to find the links, the links would be in the metadata for the packages, or accessible in some other easy way. Scraping would no longer be needed, without requiring uploads to PyPI. >> That's *their* problem. If they don't want to upload, then they don't >> want to upload. > > As the original proposal is to retain the existing behavior for already > registered/uploaded package releases (such as Twisted) so existing systems > will continue to work, but implement the suggested upload rules only for new > requests (creation/register)- so as to gradually improve the quality of PyPI > like that of other packaging systems - by encouraging authors to generate a > reasonably good sdist (setup.py + PKG-INFO) and uploading them No, that's not encouraging, that's requiring and forcing. That is NOT the same thing. Again: If you tell people you have to upload to register, the effect of that is to NOT register. It will not make anybody upload, it will make them NOT register. It has already been explained in this discussion why the Twisted folks doesn't upload to PyPI: Because it for various reasons doesn't work for them. Your solution to get more packages to PyPI is to tell people to upload or bugger off. My solution is to fix the reason they don't upload. It really is that easy: If you tell people to upload or bugger off, they will bugger off. > If I want to use a web service, I obviously have to adhere to their rules > and policies. Nobody is forcing me to do so. Exactly. Nobody is forcing anybody to use PyPI. By making it HARDER to use and have MORE requirements LESS people will use it. Is it really that hard to understand? > I assume in good faith that package authors will be happy to adapt to the > new system You are wrong. > .. for the benefit of everyone. No, its not for the benefit of everyone. It's for the benefit of adherence to random rules with no purpose. If we want more packages on PyPI, we should fix the reasons that not everyone uploads their packages. And yes, I *am* going to repeat this in different ways and wordings until your ears fall off. ;-) > Why not? Do you conceive of any reason apart from CPAN-like archives that > would help in proliferation of mirror sites and third-party sites? The point is that we *have* a CPAN like archive. > because I personally went through significant hurdles to setup a daily PyPI > mirror-like area. I just don't see how someone merely interested in writing > a third-party service, or setup a mirror of PyPI would be *most likely > inclined* to face similar hurdles before giving up. You are so focused and stuck on that before you can do anything else you have to mirror PyPI completely, only using rsync. I don't see what that would have to be so. Rsync is not the be all and end all of mirroring and most third party services do not need to mirror. They need to get data, and that's possible quite easily. >> Yes, but it's not particularly unreliable to compare the filename to >> see if it had been handled before. You don't even need to parse the >> version number for most services that work on the tarballs. > > It is indeed unreliable to rely on filenames to get package versions Yes, but it's not particularly unreliable to compare the filename to see if it had been handled before. You don't even need to parse the version number for most services that work on the tarballs. > I am not speculating as I've actually experimented with the PyPI index, > mirroring it .. handling the metadata in packages, and building it. Yes, but again, most third party service would not mirror it. A mirror would. But a mirror is only one type of third-party service out a many. >> Yes, but since thay have the source package, and will have to unpack >> it and build the packages anyway, they also have the metadata. > > It is not that simple. PyPM backend, for instance, is not monolithic as in > doing only a sequential build of packages. It first loads the dependency > graph (for which metadata - PKG-INFO/requires.txt - is required) from our > internal mirror over the network. It is expensive to go extract each and > every tarball .. from each build machine. After loading the dependency > graph, and then comparing it with existing repository .. every day, new > builds happen. You mean you build every package that also *depends* on a package that has changed? Yeah, that does require the metadata. But as I said, an easy way to mirror the metadata would definitely be an improvement. > Further, I can imagine search.cpan.org (which is not hosted by cpan.org > folks) using only the metadata without touching the source distributions. Right. Hence, they would *not* need both. Which was my point. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Fri Dec 25 10:12:26 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 10:12:26 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> References: <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> Message-ID: <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> On Fri, Dec 25, 2009 at 10:06, David Cournapeau wrote: > Which other system similar to Pypi do you know which works like Pypi > in that regard ? The whole world. Let's take this again: If you make it more complicated and have more requirements that has to be fulfilled before you can register a package on PyPI, LESS PACKAGES WILL BE REGISTERED. Again, requiring that you also upload files will not mean that everyone uploads their packages, but it will mean less packages get registered. Is this unclear? -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From cournape at gmail.com Fri Dec 25 10:33:25 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 25 Dec 2009 18:33:25 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> References: <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> Message-ID: <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> On Fri, Dec 25, 2009 at 6:12 PM, Lennart Regebro wrote: > Let's take this again: If you make it more complicated and have more > requirements that has to be fulfilled before you can register a > package on PyPI, LESS PACKAGES WILL BE REGISTERED. It is not obvious at all (that the number would be significantly less). Other systems do work even though they are much more restricting. David From regebro at gmail.com Fri Dec 25 10:55:09 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 10:55:09 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> References: <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> Message-ID: <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> On Fri, Dec 25, 2009 at 10:33, David Cournapeau wrote: > On Fri, Dec 25, 2009 at 6:12 PM, Lennart Regebro wrote: > >> Let's take this again: If you make it more complicated and have more >> requirements that has to be fulfilled before you can register a >> package on PyPI, LESS PACKAGES WILL BE REGISTERED. > > It is not obvious at all (that the number would be significantly > less). Other systems do work even though they are much more > restricting. Of course they work. That's not the point. And I'm sorry, it is obvious that it will be fewer packages registered the more work it is to register packages and the more restrictions there are on registrations. And as the benefits you want with it can be easily reached in other ways, this is the wrong path to go down. Maybe it's only obvious of you have studied some sort of social sciences or tried hard to understand management or similar things so you have gotten some basic insight into how people work as groups, but I'm not sure at all that's necessary. I really think it's completely dead obvious that the higher the requirements you have on people the more they will fail to fulfill your requirements. People are lazy. You can't demand that they aren't, you have to work with the laziness. We can't just tell package authors that they have to upload packages, you have to ask them why they do not upload them already, and then fix the problems that prevent them from uploading. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cournape at gmail.com Fri Dec 25 11:43:24 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 25 Dec 2009 19:43:24 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> References: <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> Message-ID: <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> On Fri, Dec 25, 2009 at 6:55 PM, Lennart Regebro wrote: > And I'm sorry, it is > obvious that it will be fewer packages registered the more work it is > to register packages and the more restrictions there are on > registrations. The question that matters is how significant this effect is, not that it happens. Optimizing the number of packages independently of any other criteria does not make much sense. I think many people within the group of disatisfied Pypi users would be happy to have less packages for a better overall experience. Pypi claims to have ~ 10 000 packages; I quickly counted that hackageDB has ~ 2000 packages, CRAN claims a similar number, and I think haskell community is much smaller than python (R's certainly is). > And as the benefits you want with it can be easily > reached in other ways, this is the wrong path to go down. If it were, nobody would make the argument about making things more consistent for Pypi. The goal is to make Pypi better, and easy mirroring as well as reliable experience is part of that. CRAN for example has tens if not hundred of mirrors, it works very well on all supported platforms, for people who are not necessarily programmers, and they undoubtly have much less resources than python. That's a much more convincing data point that any handwaving about so called social issues and what not, although you could argue that scientists are a particular community (but that's the one I care in the first place when I do python). David From regebro at gmail.com Fri Dec 25 12:09:03 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 12:09:03 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> References: <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> Message-ID: <319e029f0912250309y2d1921e9o7e532f630a0f4d8c@mail.gmail.com> On Fri, Dec 25, 2009 at 11:43, David Cournapeau wrote: > The question that matters is how significant this effect is, not that > it happens. Optimizing the number of packages independently of any > other criteria does not make much sense. Uploading to PyPI is so easy that almost everyone does it. Those who do not have a REASON for that. We should figure out that reason and FIX IT. You just want to say "no, we should not allow them to do that". That has *no* benefit and does not solve any problem. So there is no optimization here. This is not a question of weighing one good thing vs another conflicting good thing. > I think many people within > the group of disatisfied Pypi users would be happy to have less > packages for a better overall experience. Nobody has been able to explain why requiring file uploads would give a better overall experience. The argument for requiring uploads was that it was easier to mirror, and argument that I would claim is false in the first place. Nobody, before now, has claimed it give "an overall better experience", and I don't see how that would be the case. > If it were, nobody would make the argument about making things more > consistent for Pypi. The goal is to make Pypi better, and easy > mirroring as well as reliable experience is part of that. Yes, but uploading more file does not make mirroring of files easier. It is still a bogus argument. You can mirror PyPI files by mirroring the file structure. Yes, you will not get files that are not uploaded. But if you did, then it wouldn't be a mirror in the first place, but something else. Mirroring PyPI's metadata is apparently not very easy, and I agree that it should be. But uploading more files doesn't make mirroring of PyPI easier or harder, it just means you have more files, nothing else. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From lac at openend.se Fri Dec 25 12:22:05 2009 From: lac at openend.se (Laura Creighton) Date: Fri, 25 Dec 2009 12:22:05 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: Message from David Cournapeau of "Fri, 25 Dec 2009 19:43:24 +0900." <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> References: <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> Message-ID: <200912251122.nBPBM5pP004260@theraft.openend.se> In a message of Fri, 25 Dec 2009 19:43:24 +0900, David Cournapeau writes: >I think many people within >the group of disatisfied Pypi users would be happy to have less >packages for a better overall experience. Aside from the fact that the word you want is _fewer_ not _less_ when you are talking about a countables, I suspect this statement is true. But I don't see any relationship between 'forcing people to download their packages' and 'giving users a better experience'. And I see one major use case where forcing people to download their packages simply will not happen. Many commercial organisations have a policy that they, and only they host their own code. And they will not be willing to change this policy even if you convice them that it is in their interest to Open Source some of it, or perhaps the python bindings to some of it. How do we want to interoperate with these people and their code? Laura From ziade.tarek at gmail.com Fri Dec 25 13:12:12 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 25 Dec 2009 13:12:12 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <4B334626.6040505@v.loewis.de> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> <4B3279E9.4010507@v.loewis.de> <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> <4B3324B2.3030708@v.loewis.de> <94bdd2610912240132t345b4d8bnc1bc81d5ae82b194@mail.gmail.com> <4B334626.6040505@v.loewis.de> Message-ID: <94bdd2610912250412m4402a238j741ce6c3bc7ba53d@mail.gmail.com> 2009/12/24 "Martin v. L?wis" : [..] >>> Requires-Python: 3 >>> >>> means any 3.x, then we have 3==3.1, 3==3.2, 3.1==3.1.3, 3.2==3.2.4, >>> and, transitively, 3.1.3==3.2.4. This is undesirable. >> >> Why this is undesirable ? > > I find it undesirable if 3.1.3==3.2.4. Then, if I specify > > Requires-Python: 3.1.3 > > and I have 3.2.4 installed, it would accept that (because the > version numbers are "equal", i.e. "=="). > This looks quite complex to me. How do you translate in that case "My project is compatible with Python 3.1" ? You can't do this: Requires-Python: 3.1.1, 3.1.2, 3.1.3, etc... You could do this: Requires-Python: >=3.1,<3.2 But I find this *way* simpler: Requires-Python: 3.1 And from a reader point of view, it easily translates to : "This project requires Python 3.1" (without detailing the micro release) And if you want to say that your project requires Python 3.1.4 and up: Requires-Python: >=3.1.4 >> You'll have to convince me otherwise by explaining why it's a mistake >> to apply wildcarding when the full version is not provided. > > See above: is == transitive or not? > > I think it is counter-intuitive if the == operator on versions does > anything else than comparing version strings for string equality. > If you think that a range matching operator is needed, perhaps > a different symbol could be used (such as ~=)? Ok maybe that would be the solution indeed: if we add a ~= symbol that compares ranges, we could make it the default operator. e.g.: Requires-Python: 3.1 would mean Requires-Python: ~=3.1 wich would be equal to: Requires-Python: >=3.1,<3.2 Then == would compare precise versions. How does that sounds ? > >>>> 3.6 would include all 3.6.x releases as well. So 3.6b4 is excluded >>>> since it does not belong to the 3.6 series, but 3.6.1b4 is included. >>> Please define "belongs to the 3.6 series". In PEP 386 terminology, >>> I would expect that this means "the 'version' part is 3.6", so >>> 3.6b4 *does* belong to the 3.6 series. >> >> Sorry, here's the definition I've put behind the word belong: >> >> A version belongs to the 3.6 series if : 3.6 <= VERSION < 3.7 >> >> 3.6b4 is the fourth beta release, so : >> >> 3.6b4 < 3.6 <= VERSION < 3.7 >> >> So obviously, if the developer ask for 3.6, he doesn't want a preview of that >> version, he wants that version > > Where is that specified? I can't find it in the PEP. It is not specified like this, but the examples in PEP 386 indicates that b4 is a marker for a beta pre-release. And that "a" , "b" , "c" are post-release markers. We could add a section to explain it more precisely; > > If you really want this == operator, please add a Discussion section to > the PEP discussing that there was objection to this operator, proposing > that it should do string== instead, and that this specific semantics was > chosen because . > > Regards, > Martin > > P.S. I notice that your == definition and my explicit rewrite of it have > the same flaw: in your example, "3.7b1" would also match a version > specification of "3.6". I.e. prereleases of the next releases still fall > into the previous range. Yes. That's quite a problem. I am wondering if we should introduce a specific operator to include/exclude pre and post releases. The use case I can see would be to be able : - to exclude pre-releases and post-releases (defaut behaviour) - to include one of them, or both Regards, Tarek -- Tarek Ziad? | http://ziade.org From kiorky at cryptelium.net Fri Dec 25 13:27:20 2009 From: kiorky at cryptelium.net (kiorky) Date: Fri, 25 Dec 2009 13:27:20 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> <4B335048.4060707@cryptelium.net> <319e029f0912240556g34cde9b5qce81faec5a6ea081@mail.gmail.com> <4B33DB25.7010608@cryptelium.net> Message-ID: <4B34AFA8.4050504@cryptelium.net> Tres Seaver a ?crit : > kiorky wrote: > I would say that having a package author *not* upload the distributions > is their right, but I would likely avoid using such a package, That depend, people can not upload their packages because previous bad experience, for false generated sdist for example. > just on that basis. Note that I build per-project mirrors of the pacakges I use > anyway, in part not to depend on either PyPI You depend on them anyway in first place anyway, at the first installation, even in dev or pre-production modes. And having problems at those stages have maybe less drawbacks but you are nevertheless blocked. Having a single archive which supports mirrors "officially" would just be safer than a single archived not officially mirrored with thirdparty satellite mirrors which can be randomly down. And having Personal/Corporate PyPi/eggs mirrors are beyond the scope, here, i think. It's just an additional and mandatory security policy to deploy projects nowodays. > or other download sources > for supporting apps in production: I just prefer to use only > freely-distributable software. As, i think, mostly of us including me. And 99,9% softwares registered on Pypi. So, comes my idea that we would have just to get the source distributions where they are no matter how they would have been generated and mirror them as-is on Pypi which could be the only thing to mirror (and i don't say here that mirroring pypi is synomym of easy, Lennart) to get a bit safer. In a nowodays projet, i get often errors with thirdparty mirrors. It may be just bad chance, but i got problems. > Tres. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From regebro at gmail.com Fri Dec 25 14:35:34 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 14:35:34 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B34AFA8.4050504@cryptelium.net> References: <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <319e029f0912240244g4f2ec3a3m71735c80d338f7b3@mail.gmail.com> <4B335048.4060707@cryptelium.net> <319e029f0912240556g34cde9b5qce81faec5a6ea081@mail.gmail.com> <4B33DB25.7010608@cryptelium.net> <4B34AFA8.4050504@cryptelium.net> Message-ID: <319e029f0912250535s543a23f4l59e5e9a65afc887@mail.gmail.com> On Fri, Dec 25, 2009 at 13:27, kiorky wrote: > So, comes my idea that we would have just to get the source distributions where > they are no matter how they would have been generated and mirror them as-is on > Pypi which could be the only thing to mirror (and i don't say here that > mirroring pypi is synomym of easy, Lennart) to get a bit safer. For clarification: I think this is a good idea. I notice the complete lack of requiring uploads, or for that matter requiring anything out of the package authors. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From sridharr at activestate.com Fri Dec 25 19:30:34 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Fri, 25 Dec 2009 10:30:34 -0800 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912250109r63f44df2i679e5e06dac5e875@mail.gmail.com> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <4B347129.1060106@activestate.com> <319e029f0912250109r63f44df2i679e5e06dac5e875@mail.gmail.com> Message-ID: <4B3504CA.1070600@activestate.com> On 12/25/2009 1:09 AM, Lennart Regebro wrote: >> Why not? Do you conceive of any reason apart from CPAN-like archives that >> would help in proliferation of mirror sites and third-party sites? > > The point is that we *have* a CPAN like archive. I am amazed at the amount of denial in this discussion, and I'll pass. I have nothing further to discuss. -srid From regebro at gmail.com Fri Dec 25 19:55:42 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 25 Dec 2009 19:55:42 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B3504CA.1070600@activestate.com> References: <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <4B344209.3090209@activestate.com> <319e029f0912242227h22b1b33bw1dc291f818c8118f@mail.gmail.com> <4B347129.1060106@activestate.com> <319e029f0912250109r63f44df2i679e5e06dac5e875@mail.gmail.com> <4B3504CA.1070600@activestate.com> Message-ID: <319e029f0912251055y39cfe47fl4e68a6038d1a6586@mail.gmail.com> On Fri, Dec 25, 2009 at 19:30, Sridhar Ratnakumar wrote: > I am amazed at the amount of denial in this discussion, and I'll pass. Explanations are not denial. Several concrete points where PyPI can be improved has come forward. Requiring upload is not one of those and will not make it better, even if it is possible it makes it more CPAN-like. Neither is the requirement that metadata must be available in the same file structure as the distribution files something that will magically make PyPI into CPAN. The idea that to be as good as CPAN is has to be exactly like CPAN and work exactly like CPAN is misguided. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From martin at v.loewis.de Fri Dec 25 22:58:29 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 25 Dec 2009 22:58:29 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> <4B333D3B.7010403@cryptelium.net> <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> <4B334068.2060900@cryptelium.net> <94bdd2610912240244q385fc4ebg7d86aeb19610afbe@mail.gmail.com> Message-ID: <4B353585.5020404@v.loewis.de> >>> But it's only butter, in fact i am just happy with sdist upload. >> Although SSH is quite a heavy development on PyPI side, it means we >> would have to implement >> an SSH server. (like Zope did I think for their development server, >> using Paramiko IIRC) > > cvs.zope.org / svn.zope.org (same machine) run a stock sshd: they use > the "command=" prefix on users' pubkeys to limit what that key can be > used to do (only SVN / CVS operations for any non-admin users). That works well because both cvs and subversion have hard-coded support for a remote server application, along with a proprietary protocol. Adding that kind of protocol to an application that is primarily based on http is not straight-forward (it can be done, of course). Regards, Martin From martin at v.loewis.de Fri Dec 25 23:04:47 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 25 Dec 2009 23:04:47 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <87fx706o7z.fsf@benfinney.id.au> References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <4B327B2C.4080605@v.loewis.de> <4B32894B.8090109@activestate.com> <4B33276E.2070900@v.loewis.de> <87oclo7mgc.fsf@benfinney.id.au> <4B334273.1050707@v.loewis.de> <87k4wc7eh5.fsf@benfinney.id.au> <4B336D08.8060308@v.loewis.de> <87fx706o7z.fsf@benfinney.id.au> Message-ID: <4B3536FF.9020707@v.loewis.de> >> Ben Finney wrote: >>> That isn't a good argument. By the same logic, PyPI should not >>> reject *any* upload, to avoid ?forcing? uploaders to do extra work. >> PyPI's rejection of certain uploads is primarily to prevent spam from >> being uploaded. > > Am I wrong, then, in thinking that PyPI will reject an upload with > malformed metadata? I don't want to elaborate the specifics in something that gets archived. If you want to know what precise checks are performed, read the source code. In principle, yes, you are wrong: PyPI may accept uploads even if the metadata are malformed. > To my understanding, this discussion is about arguing whether an upload > that is missing the package should be rejected by PyPI. Ah. In PyPI, there are two kinds of "write" interactions: "registration", and "upload". The latter is what brings distribution files on PyPI. This is the one I'm talking about. The registration is *only* about metadata (i.e. no files at all), and it does check the consistency of the metadata. Regards, Martin From kiorky at cryptelium.net Fri Dec 25 23:05:01 2009 From: kiorky at cryptelium.net (kiorky) Date: Fri, 25 Dec 2009 23:05:01 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B353585.5020404@v.loewis.de> References: <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <4B3298D7.6040706@cryptelium.net> <4B33290A.80809@v.loewis.de> <4B333D3B.7010403@cryptelium.net> <94bdd2610912240214g6a5c8105o5d8b392b4bbec45c@mail.gmail.com> <4B334068.2060900@cryptelium.net> <94bdd2610912240244q385fc4ebg7d86aeb19610afbe@mail.gmail.com> <4B353585.5020404@v.loewis.de> Message-ID: <4B35370D.2050005@cryptelium.net> Martin v. L?wis a ?crit : >>> Although SSH is quite a heavy development on PyPI side, it means we >>> would have to implement >>> an SSH server. (like Zope did I think for their development server, >>> using Paramiko IIRC) >> cvs.zope.org / svn.zope.org (same machine) run a stock sshd: they use >> the "command=" prefix on users' pubkeys to limit what that key can be >> used to do (only SVN / CVS operations for any non-admin users). > That works well because both cvs and subversion have hard-coded support > for a remote server application, along with a proprietary protocol. > Adding that kind of protocol to an application that is primarily based > on http is not straight-forward (it can be done, of course). Additionnal to limit via the command="" prefix, making ssh wrapper scripts to allow a subset of commands or using simple things like "rssh" is really simple to do to just allow controlled access. We are not obliged to make the application aware of the underlying ssh infra. For example, we can upload our packages somewhere on 'the host' using plain scp and we can have other mecanisms to load them in the pypi database. > Regards, > Martin -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From martin at v.loewis.de Fri Dec 25 23:18:25 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 25 Dec 2009 23:18:25 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B3448F0.4090808@activestate.com> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <20091224104658.GA12434@rivendell.internal> <4B3349B5.3000804@v.loewis.de> <4B3448F0.4090808@activestate.com> Message-ID: <4B353A31.5070700@v.loewis.de> > It is unreliable [bugs.launchpad.net/pypi-mirror/+bug/386143] and lacks > the pre-extracted metadata. I wouldn't call it a mirror tool, for it is > not an exact copy of PyPI data[1]. > > *** > [1] "In computing, a mirror is an exact copy of a data set. On the > Internet, a mirror site is an exact copy of another Internet site. ..." In this strict sense, PyPI will never have a mirror, nor does any of the other project hosting services have a mirror in the strict sense. In particular, I'm not willing (and not allowed to) give mirrors access to PyPI's user database: account names and email addresses. Regards, Martin From martin at v.loewis.de Fri Dec 25 23:28:17 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 25 Dec 2009 23:28:17 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <94bdd2610912250412m4402a238j741ce6c3bc7ba53d@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> <4B3279E9.4010507@v.loewis.de> <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> <4B3324B2.3030708@v.loewis.de> <94bdd2610912240132t345b4d8bnc1bc81d5ae82b194@mail.gmail.com> <4B334626.6040505@v.loewis.de> <94bdd2610912250412m4402a238j741ce6c3bc7ba53d@mail.gmail.com> Message-ID: <4B353C81.1080702@v.loewis.de> > Ok maybe that would be the solution indeed: if we add a ~= symbol > that compares ranges, we could make it the default operator. e.g.: > > Requires-Python: 3.1 > > would mean > > Requires-Python: ~=3.1 > > wich would be equal to: > > Requires-Python: >=3.1,<3.2 > > > Then == would compare precise versions. > > > How does that sounds ? Maybe it's bike-shedding, but yes, I would like to have a separate operator for this range matching. So it would be fine with me. >> Where is that specified? I can't find it in the PEP. > > It is not specified like this, but the examples in PEP 386 indicates > that b4 is a marker for a beta pre-release. And that "a" , "b" , "c" > are post-release markers. I agree that PEP 386 specifies sufficiently that 3.6b4 < 3.6. However, I was wondering where it is specified that your == operator is equivalent to a range starting at the value. If it was "wildcard matching" (which I assume it was), I would have expect that "==3.6" means "matches 3\.6.*" (in re syntax), so 3.6b4 would match. > Yes. That's quite a problem. I am wondering if we should introduce > a specific operator to include/exclude pre and post releases. > > The use case I can see would be to be able : > - to exclude pre-releases and post-releases (defaut behaviour) > - to include one of them, or both It may be useful, but I would wait for user demand. In the end, if you install a prerelease, and then find that something breaks, it's really "your fault". Regards, Martin From cournape at gmail.com Sat Dec 26 05:03:32 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 26 Dec 2009 13:03:32 +0900 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <200912251122.nBPBM5pP004260@theraft.openend.se> References: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> Message-ID: <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> On Fri, Dec 25, 2009 at 8:22 PM, Laura Creighton wrote: > In a message of Fri, 25 Dec 2009 19:43:24 +0900, David Cournapeau writes: >>I think many people within >>the group of disatisfied Pypi users would be happy to have less >>packages for a better overall experience. > > Aside from the fact that the word you want is _fewer_ not _less_ when > you are talking about a countables, I suspect this statement is true. Sorry about that, I don't always double check my English. > But I don't see any relationship between 'forcing people to download > their packages' and 'giving users a better experience'. I guess you meant upload. The reasons I see for making some things, in particular upload mandatory are as follows: - file upload makes pure rsync-based mirroring. In the case of CRAN, you mirror with one rsync command. No need for fancy scheme, new packages, etc... - lack of tarballs/installers means that when you use an installer, for example easy_install, it has to find it in another way. This way is based on scraping: the website may be dead, not available anymore, etc... In that case, installation fails. Different websites may also have different limitations (think about corporate networks). - more generally, if there is an information missing on Pypi, be it in the form of metadata, data, etc..., it means it will have to be inferred back. Making upload easier seems a very weak argument to justify this. > Many commercial organisations have a policy that they, and only they > host their own code. ?And they will not be willing to change this policy > even if you convice them that it is in their interest to Open Source > some of it, or perhaps the python bindings to some of it. ?How do we > want to interoperate with these people and their code? I did not know it was even "allowed" to register non open source code in Pypi. That's the first time I see the argument given (and the first time a real argument is given). Is this a major requirement ? David From regebro at gmail.com Sat Dec 26 09:18:40 2009 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 26 Dec 2009 09:18:40 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> References: <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> Message-ID: <319e029f0912260018g32dc3195vef58f59e4b37edf6@mail.gmail.com> On Sat, Dec 26, 2009 at 05:03, David Cournapeau wrote: > I guess you meant upload. The reasons I see for making some things, in > particular upload mandatory are as follows: > ?- file upload makes pure rsync-based mirroring. In the case of CRAN, > you mirror with one rsync command. No need for fancy scheme, new > packages, etc... But you can already do that with the files that are uploaded. You claim that to mirror PyPI, you have to *also* include files that are *not* hosted on PyPI. That makes no sense. Sorry. > ?- lack of tarballs/installers means that when you use an installer, > for example easy_install, it has to find it in another way. Which is already does, do that's not a problem. > This way is based on scraping I agree that it would be better if the package uploaders included a download URL in the metadata. This for some reason seems unusual. > the website may be dead, not available anymore, > etc... In that case, installation fails. This is true. I agree it would be better of people uploaded their packages to PyPI. But this is a difference in attitude, and it's a difference between forcing people to upload, and helping people. It really is what it comes down to. I tried to avoid saying it in such plain terms, but the message seems not to reach through. I think we should figure out *why* people are not uploading and then help them fix the reason this is not so. You say we should tell them to upload or bugger off. That will mean that people buggers off. That is still NOT an improvement, and it will NOT become an improvement because you repeat it more times. Requiring uploads will make PyPI *worse* not better and means it will have *less* packages, and when you can't even find a package via a download URL or scraping, well, then you can't find it automatically at all. > I did not know it was even "allowed" to register non open source code > in Pypi. ?That's the first time I see the argument given (and the > first time a real argument is given). Is this a major requirement ? Since I have given exactly that argument: That licenses and similar prevents people from uploading some code, that is proof that you don't read my answers. You should. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From eric at trueblade.com Sat Dec 26 11:13:37 2009 From: eric at trueblade.com (Eric Smith) Date: Sat, 26 Dec 2009 05:13:37 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <319e029f0912260018g32dc3195vef58f59e4b37edf6@mail.gmail.com> References: <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <319e029f0912260018g32dc3195vef58f59e4b37edf6@mail.gmail.com> Message-ID: <4B35E1D1.8000204@trueblade.com> Lennart Regebro wrote: > On Sat, Dec 26, 2009 at 05:03, David Cournapeau wrote: >> The reasons I see for making some things, in >> particular upload mandatory are as follows: >> - file upload makes pure rsync-based mirroring. In the case of CRAN, >> you mirror with one rsync command. No need for fancy scheme, new >> packages, etc... > > But you can already do that with the files that are uploaded. You > claim that to mirror PyPI, you have to *also* include files that are > *not* hosted on PyPI. That makes no sense. Sorry. When some people say "I want to mirror PyPI", I think they mean "I want to have a private copy of all of the files I need to install a given set of packages I need". This is often a requirement because the computer(s) where an installation is occurring are behind firewalls without general Internet access or are production servers where administrative rules require that all installation repositories be locked down. (I'm not saying that David is using mirroring in this sense, but I know some people, including me, do use it that way.) Actually mirroring PyPI (modulo Martin's issues with accounts, etc.) can be achieved with rsync or other ways of just copying the files on PyPI. Producing a private PyPI with all of the packages you need for installation is less straight forward when all of the files are not uploaded to PyPI. I usually do it by trial and error, but I've not spent any time looking into automated solutions. Eric. From smueller at cpan.org Sat Dec 26 11:17:20 2009 From: smueller at cpan.org (Steffen Mueller) Date: Sat, 26 Dec 2009 10:17:20 +0000 (UTC) Subject: [Distutils] =?utf-8?q?Python_people_want_CPAN_and_how_the_latter_?= =?utf-8?q?came=09about?= References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <319e029f0912241514y1e818590xf8cb9d52dd9ed477@mail.gmail.com> Message-ID: Hi Lennart, Lennart Regebro gmail.com> writes: > On Wed, Dec 23, 2009 at 10:32, S. Mueller wrote: > >> I have to say that I vastly prefer not to have any authorization and > >> allow anyone to release anything in any namespace. But then I am > >> getting fanatically anarchical in these issues. You can not organize > >> freedom. > > > > But you have to. This is clearly a case of citation rape. ;) > No, I really mean it whan I say you can't. And you never *have* to do > the impossible, and trying just leads to problems. I realize this is a > matter of attitude, but if the sentence "I want CPAN" means "I want > restrictions and controls and people preventing others from uploading > stuff", then they are misguided. Sorry, but I'm not being philosophical when I say you have to authorize access to things. Apparently the Python repository does, too. Or otherwise I'll upload a few popular packages with high version numbers that contain viruses for New Year. I'm not talking about restricting freedom. Now, on CPAN, I *can* upload anything even if not authorized to do so. It just won't be part of the official indexes if I upload a new version of the database interface DBI. > > What you're saying here means you virtually throw > > away the ability to do anything useful with namespace meta data. > > *shrugs* Namespaces has no metadata in Python. They are just > namespaces, no more, no less. The names of your *packages* is > protected on PyPI. But several people can use the same *namespace*. > Ie, nobody can upload a "Twisted" package, except those who have the > permission. But people can upload a "zope.my.own.package", even though > the namespace "zope" is already used by other packages. Same for CPAN. You automatically register an exact namespace by uploading a file that contains it. But you don't get it recursively. Please recall my explanation of how in Perl a namespace == package name == class name. If I upload Steffens::Module, somebody else can upload Steffens::Module::Plugin. Just Steffens::Module is restricted from the point on of the initial upload. That we do out meta data stuff on package/namespace/class names as opposed to distribution names has the huge benefit of interoperability between distributions. > > Think about it like this: If you install any module from CPAN (and > > only the valid ones end up in the index), you can use all of them in > > the same application. If module A and B could both implement > > Config::Parser, then you couldn't use both of them at the same time. > > This would be true for Python too. But Python doesn't try to pretend > that all the packages that exist are some sort of standard library, > and therefore don't try to put them all into one sort of hierarchical > organized namespace. And to be honest I don't see the point of doing > that. We're not pretending anything. We're not forcing anything except that you don't override somebody elses work. We advise on proper choice of namespaces. But in the end, we never force anybody to adhere to our preferences. By "our", I mean an arbitrary bunch of experienced contributors who offer advice for new contributors. Most of those wouldn't even have the power to impose any restrictions. Those who do, use it only extremely sparely. For example, when somebody has passed away or simply asks for help in passing maintainership to someone else. > > Still. We allow for a lot of creative freedom. We just don't want a > > random newbie upload a shiny package "DB" which implements his idea of > > a database interface when it's really the name of the package that > > implements the Perl debugger. He can still upload. The file will be > > accessible in his CPAN directory. Users can install it from the CPAN > > shell as "install NEWBIEUSERID/DB-1.00.tar.gz", but not as "install > > DB" or "$ cpan DB". > > I see. But IMO Perl then there starts out with trying to organize > freedom from the start, and that then leads to the above problem that > newbies can come up and mess up this so called organized freedom, > which means you need to restrict it even more by having people control > and restrict the namespaces, etc, etc. You end up having to have more > organisation to fix the problems your organisation caused. This is, > without trying to be rude or anything", the fate of all bureaucracies. You're wrong. The Perl/PAUSE/CPAN system works exceptionally well. But it does so because we regulate a lot less than you think. Let me recap the major points once more: a) Anybody can upload anything (theoretically, even you wedding pictures) a1) If it's a virus and identified, it'll be deleted. a2) Anything else goes into CPAN. b) PAUSE extracts archives and checks whether it looks like a Perl module distribution. b1) If not, it's simply ignored (but still mirrored) b2) If so, it's scanned for Perl classes. (== namespaces == packages) c) The classes found are added to the official index. c1) This is the case if the classes have never been uploaded before. c2) Also, if *you* uploaded them before. c3) Or if the guy who first did transfered maintenance or assigned you as co-maintainer. c4) If somebody else offers a class named $foo, your distribution enters CPAN, but doesn't get included in the indexes until the clash has been resolved. Now, c4) takes care of newbies "messing up the organized freedom". It's a safeguard for users who expect DBI.pm to always be Tim Bunce's DBI.pm. But it might as well be a bit of legal protection, too. > > This is a safeguard against insanity and it's the thing that means > > that you can trust "cpan PAR" to always install the Perl Archive > > Toolkit that was released by Autrijus Tang, Roderich Schupp, or myself > > (we share co-maintenance). It's never going to be some random junk. > > And that you auto-register a namespace on upload is the guarantee. > > And obviously on PyPI, it's first come first serve as well. But nobody > would call a db package "db" if one already exists. Why would they do > that? What's the point? Why would I make a completely new package > called "Twisted" for example? There already is one. It's just a > mindset that is completely incomprehensible to me. Then you clearly do not understand what it is like to be a) malicious b) new, young, inexperienced c) stupid. > I expect what I would call creative freedom, you would call total anarchy. No. But you did yourself in an earlier mail. :) > > PS: Let me comment on another post of yours quickly. No. In the Perl > > community, the name "CPAN" doesn't yield confusion. It's just a way to > > refer to the whole ecosystem > > OK, that's not how it sounded in your first post, thanks for clarifying. Sorry if I caused confusion. Best regards, Steffen From regebro at gmail.com Sat Dec 26 11:47:43 2009 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 26 Dec 2009 11:47:43 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <319e029f0912241514y1e818590xf8cb9d52dd9ed477@mail.gmail.com> Message-ID: <319e029f0912260247k16aacc50p7b9eaa83e8ae1770@mail.gmail.com> On Sat, Dec 26, 2009 at 11:17, Steffen Mueller wrote: > This is clearly a case of citation rape. ;) Possibly, but I tried to extract the essence of the misunderstanding. Maybe I was boiling too hard. :) > Sorry, but I'm not being philosophical when I say you have to authorize access > to things. But the discussion was not "things". The discussion was specifically *namespaces*. > Same for CPAN. You automatically register an exact namespace by uploading a file > that contains it. But you don't get it recursively. Please recall my explanation > of how in Perl a namespace == package name == class name. It isn't in Python. > That we do out meta data stuff on package/namespace/class names as opposed to > distribution names has the huge benefit of interoperability between > distributions. What problems would that be? > We're not pretending anything. We're not forcing anything except that you don't > override somebody elses work. We advise on proper choice of namespaces. But in > the end, we never force anybody to adhere to our preferences. By "our", I mean > an arbitrary bunch of experienced contributors who offer advice for new > contributors. Most of those wouldn't even have the power to impose any > restrictions. Those who do, use it only extremely sparely. For example, when > somebody has passed away or simply asks for help in passing maintainership to > someone else. Then there simply is no difference between PyPI and CPAN in this regard. > You're wrong. The Perl/PAUSE/CPAN system works exceptionally well. But it does > so because we regulate a lot less than you think. The question then becomes why it is claimed that PyPI needs to regulate *more* when it's pretty obvious once the confusion has cleared that the only regulation on both systems is that you can't upload a new version of somebody else's package. It's just that you call packages namespaces and we don't. (Or well, on CPAN it canbe uploaded, but it won't be visible, in PyPI it can't be uploaded). >> > This is a safeguard against insanity and it's the thing that means >> > that you can trust "cpan PAR" to always install the Perl Archive >> > Toolkit that was released by Autrijus Tang, Roderich Schupp, or myself >> > (we share co-maintenance). It's never going to be some random junk. >> > And that you auto-register a namespace on upload is the guarantee. >> >> And obviously on PyPI, it's first come first serve as well. But nobody >> would call a db package "db" if one already exists. Why would they do >> that? What's the point? Why would I make a completely new package >> called "Twisted" for example? There already is one. It's just a >> mindset that is completely incomprehensible to me. > > Then you clearly do not understand what it is like to be > a) malicious > b) new, young, inexperienced > c) stupid. So? What does that have to do with anything? Why would PyPI need more regulation/organisation to handle that? Especially if CPAN doesn't? As far as I can see, this is another case of: "Python needs CPAN!" "We have it already." :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From regebro at gmail.com Sat Dec 26 11:52:50 2009 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 26 Dec 2009 11:52:50 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B35E1D1.8000204@trueblade.com> References: <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <319e029f0912260018g32dc3195vef58f59e4b37edf6@mail.gmail.com> <4B35E1D1.8000204@trueblade.com> Message-ID: <319e029f0912260252w3f1893e1sd96793048c3867cd@mail.gmail.com> On Sat, Dec 26, 2009 at 11:13, Eric Smith wrote: > When some people say "I want to mirror PyPI", I think they mean "I want to > have a private copy of all of the files I need to install a given set of > packages I need". This is often a requirement because the computer(s) where > an installation is occurring are behind firewalls without general Internet > access or are production servers where administrative rules require that all > installation repositories be locked down. Right, setting up your *own package index* is not *mirroring*. You do not need to mirror all of PyPI, and you do not need to mirror the PyPI metadata do do this. And the argument of file uploads has been used for mirroring as well, namely for third-party services. And that argument still does not hold water. Setting up package indexes has benefits and there are as mentioned several different ways, one being a mirrored package index, ie caching the other setting up special indexes for different versions of the software etc. Yes, it's correct, if people would just add download URLs or upload the packages on PyPI, downloading all the files to make your own package index would be somewhat easier. Is this really a major difference to CPAN? If people don't upload a package to CPAN, what do you do? Does this get magically resolved somehow? I don't see how that would work. * If you only use packages on PyPI, it's easy. * If you only use packages on CPAN, it's easy. * If you use packages not on PyPI, it's more complicated. * If you use packages not on CPAN, it's more complicated. Seems to be the same to me. And it ends with the same conclusion: We must figure WHY people don't upload to PyPI and FIX THEIR PROBLEM. Saying "Upload or bugger off" does NOT solve this problem. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From martin at v.loewis.de Sat Dec 26 11:55:49 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 26 Dec 2009 11:55:49 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B35E1D1.8000204@trueblade.com> References: <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <319e029f0912260018g32dc3195vef58f59e4b37edf6@mail.gmail.com> <4B35E1D1.8000204@trueblade.com> Message-ID: <4B35EBB5.6050605@v.loewis.de> > When some people say "I want to mirror PyPI", I think they mean "I want > to have a private copy of all of the files I need to install a given set > of packages I need". This is often a requirement because the computer(s) > where an installation is occurring are behind firewalls without general > Internet access or are production servers where administrative rules > require that all installation repositories be locked down. For this specific need, automated solutions do exist, allowing either explicit specification of the packages that you want to have available, or transparently caching all the ones that you happened to access. Regards, Martin From kiorky at cryptelium.net Sat Dec 26 12:14:04 2009 From: kiorky at cryptelium.net (kiorky) Date: Sat, 26 Dec 2009 12:14:04 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B353A31.5070700@v.loewis.de> References: <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <4B326E7E.3050407@activestate.com> <319e029f0912231333i2505253qebcc6c60f7037d8d@mail.gmail.com> <94bdd2610912231432y5168fcf7ye477ff294a88ff5d@mail.gmail.com> <20091224025928.15596.524671682.divmod.xquotient.507@localhost.localdomain> <20091224104658.GA12434@rivendell.internal> <4B3349B5.3000804@v.loewis.de> <4B3448F0.4090808@activestate.com> <4B353A31.5070700@v.loewis.de> Message-ID: <4B35EFFC.7050402@cryptelium.net> Martin v. L?wis a ?crit : >> It is unreliable [bugs.launchpad.net/pypi-mirror/+bug/386143] and lacks >> the pre-extracted metadata. I wouldn't call it a mirror tool, for it is >> not an exact copy of PyPI data[1]. >> >> *** >> [1] "In computing, a mirror is an exact copy of a data set. On the >> Internet, a mirror site is an exact copy of another Internet site. ..." > > In this strict sense, PyPI will never have a mirror, nor does any of the > other project hosting services have a mirror in the strict sense. > > In particular, I'm not willing (and not allowed to) give mirrors access > to PyPI's user database: account names and email addresses. Mirrors are just mirrors, We can strip down the pypi database to bare-mirroring minimum. Mirrors could just be read-only. Thus by deleting or changing all personal accounts and other critical informations to something "random". We do not need complete pypi database on mirrors. It's largely enough to get the distributions from and to install things. It doesn't require also much developments to get it ready. Just the script that make the current database "informationless" and "download safe" and limited access to the filesystem where are stored the distributions for replication. We can imagine some xmlrpc mecanisms to synchronise mirrors databases without implicating to change the mirror database after some time. I didn't look to existing pypimirroring tools, i think some implemention (z3c.pypimirror at least) must do something in that way and could be a good starter. > Regards, > Martin -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From martin at v.loewis.de Sat Dec 26 12:22:01 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 26 Dec 2009 12:22:01 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <319e029f0912241514y1e818590xf8cb9d52dd9ed477@mail.gmail.com> Message-ID: <4B35F1D9.4010400@v.loewis.de> > Sorry, but I'm not being philosophical when I say you have to authorize access > to things. Apparently the Python repository does, too. Or otherwise I'll upload > a few popular packages with high version numbers that contain viruses for New > Year. Maybe it's a terminology matter. "I" have to "authorize": who is "I"? In PyPI, no person ever authorizes access. Yet, you still cannot upload newer versions of popular packages. Package names are registered on a first-come first-served basis. You could register a popular package if you were the first one to do so. However, all the popular packages are already registered, and then only the person originally performing the registration may upload newer versions (strictly speaking, that person can also delegate that permission to others, but that is besides the point that the archive operators will never need to authorize anything). > Now, on CPAN, I *can* upload anything even if not authorized to do so. It just > won't be part of the official indexes if I upload a new version of the database > interface DBI. In PyPI, you can upload the popular package under a different name. It will be indexed and all, but people still may not use it because of the different name. > That we do out meta data stuff on package/namespace/class names as opposed to > distribution names has the huge benefit of interoperability between > distributions. I think you misunderstood the Python term "distribution" here (or I misunderstand the point you make). A "distribution" is a tar file (or such); it's what library authors distribute. There can't be "interoperability" between them, at least not in the way I understand "interoperability". >>> Think about it like this: If you install any module from CPAN (and >>> only the valid ones end up in the index), you can use all of them in >>> the same application. If module A and B could both implement >>> Config::Parser, then you couldn't use both of them at the same time. >> This would be true for Python too. But Python doesn't try to pretend >> that all the packages that exist are some sort of standard library, >> and therefore don't try to put them all into one sort of hierarchical >> organized namespace. And to be honest I don't see the point of doing >> that. > > We're not pretending anything. We're not forcing anything except that you don't > override somebody elses work. I think the point here was that "we" see the advantage that CPAN imposes with the namespace registrations primarily as theoretical. Yes, it does prevent two people putting code into Config::Parser, and yes, in theory, it may be that they do the same in Python with PyPI. In practice, that is never a problem - there are so many names to chose from, and if you do happen to conflict with somebody else's naming choice, AND there is indeed interest in using both packages simultaneously, your users will ask you to rename your package. However, that happens so rarely if at all that it hasn't been a real concern. >> And obviously on PyPI, it's first come first serve as well. But nobody >> would call a db package "db" if one already exists. Why would they do >> that? What's the point? Why would I make a completely new package >> called "Twisted" for example? There already is one. It's just a >> mindset that is completely incomprehensible to me. > > Then you clearly do not understand what it is like to be > a) malicious > b) new, young, inexperienced > c) stupid. If you have malicious users, and unsuspecting users download and run their code, no namespacing mechanism can stop them, neither in CPAN nor in PyPI. Malicious users would be able to bypass any checks that are performed, and experienced users, code review etc. needs to discover that. New or stupid users may accidentally create colliding names. However, in Python, packages aren't called "db", but, say, "psycopg2", "Twisted", "django", or "zope". Chances are fairly low that new users accidentally come up with such a name. Regards, Martin From lac at openend.se Sat Dec 26 13:15:14 2009 From: lac at openend.se (Laura Creighton) Date: Sat, 26 Dec 2009 13:15:14 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: Message from David Cournapeau of "Sat, 26 Dec 2009 13:03:32 +0900." <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> References: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> Message-ID: <200912261215.nBQCFEYH026480@theraft.openend.se> In a message of Sat, 26 Dec 2009 13:03:32 +0900, David Cournapeau writes: >I guess you meant upload. The reasons I see for making some things, in >particular upload mandatory are as follows: > - file upload makes pure rsync-based mirroring. In the case of CRAN, >you mirror with one rsync command. No need for fancy scheme, new >packages, etc... > - lack of tarballs/installers means that when you use an installer, >for example easy_install, it has to find it in another way. This way >is based on scraping: the website may be dead, not available anymore, >etc... In that case, installation fails. Different websites may also >have different limitations (think about corporate networks). This, of course, is one reason why some people want to do exactly this. Right now I don't know any way to say 'under no circumstances, ever, let easy_install near my code because it will do very bad things to it'. > - more generally, if there is an information missing on Pypi, be it >in the form of metadata, data, etc..., it means it will have to be >inferred back. Making upload easier seems a very weak argument to >justify this. Ah, this isn't the argument about laziness and easy of use. This is the argument about control. > >> Many commercial organisations have a policy that they, and only they >> host their own code. ??And they will not be willing to change this policy >> even if you convice them that it is in their interest to Open Source >> some of it, or perhaps the python bindings to some of it. ??How do we >> want to interoperate with these people and their code? > >I did not know it was even "allowed" to register non open source code >in Pypi. That's the first time I see the argument given (and the >first time a real argument is given). Is this a major requirement ? It's not about non-open source code. It's about code that is open sourced, but still copyright the company that made it. The company may have strict rules about where it hosts its code, for reasons of guarding its reputation, protection from lawsuits, and the like. This doesn't stop somebody else from taking their code and then uploading it to pypi, of course, but that's not what the corporate mandate says. And, with pypi adding more features, and starting to branch into customer support, and bug tracking, and a rating system and the whole social networking bit ... there is more and more reasons for companies to decide that the burden of being part of the 'pypi whole experience' doesn't justify making the upload. I liked things a whole lot better when pypi was about being a package index, and _only_ about being a package index, and where those people who had ideas about improving the user experience were free to go out there and write their own programs to do the same, but where none of these has any sort of 'official recognition' and where, of course, others who didn't want that sort of experience were free to ignore the whole thing. Laura > >David From regebro at gmail.com Sat Dec 26 13:41:03 2009 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 26 Dec 2009 13:41:03 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <200912261215.nBQCFEYH026480@theraft.openend.se> References: <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <200912261215.nBQCFEYH026480@theraft.openend.se> Message-ID: <319e029f0912260441j2d11507co2cda5287b1031b51@mail.gmail.com> On Sat, Dec 26, 2009 at 13:15, Laura Creighton wrote: > This, of course, is one reason why some people want to do exactly > this. ?Right now I don't know any way to say 'under no circumstances, > ever, let easy_install near my code because it will do very bad things > to it'. Well.... Not registering on PyPI? :-) Another way is to not include a setup.py. easy_install will then not try to run setup.py install. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ssteinerx at gmail.com Sat Dec 26 16:48:28 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sat, 26 Dec 2009 10:48:28 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <200912261215.nBQCFEYH026480@theraft.openend.se> References: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <200912261215.nBQCFEYH026480@theraft.openend.se> Message-ID: <51069813-92D4-4769-9939-1B6B94B7F4D9@gmail.com> On Dec 26, 2009, at 7:15 AM, Laura Creighton wrote: > Right now I don't know any way to say 'under no circumstances, > ever, let easy_install near my code because it will do very bad things > to it'. Uh...I think you just did. > I liked things a whole lot better when pypi was about being a package > index, and _only_ about being a package index, and where those people > who had ideas about improving the user experience were free to go out > there and write their own programs to do the same, but where none of > these has any sort of 'official recognition' and where, of course, > others who didn't want that sort of experience were free to > ignore the whole thing. I think that, in the whole CPAN-ification of PyPI discussion, an absurd amount feature creep has come into the discussion. I think the ratings discussion was the tiny crystal that started the whole gigantic snowball. At the bottom of everything CPAN's repository is just a glorified, rsync-able FTP site with a bunch of stuff in directories. Everything on top of that is window dressing. The PyPI discussions seem to be tending toward mixing the window dressing with the framing, to use a building analogy, and what that will result in is a weak frame and ugly windows. A building that slowly (or quickly) falls down under its own weight, and looks bad doing it. I think that splitting > package storage and pointers to off-repository storage (for those who don't upload to PyPI) > metadata about the stored packages > tools for creating stored packages > tools for retrieving stored packages > tools for installing packages would go a long way towards unobfuscating this whole discussion. Yes, I'm sure someone will disagree with some fine-point of that division but isn't that what woodshedding is all about? S From lregebro at jarn.com Sat Dec 26 18:40:46 2009 From: lregebro at jarn.com (Lennart Regebro) Date: Sat, 26 Dec 2009 18:40:46 +0100 Subject: [Distutils] How Python can have CPAN. Message-ID: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> I have looked through the long discussion about the topic. As far as I can see these are the sensible, reasonable, concrete suggestions of what can be done to improve PyPI towards CPAN quality: 1. Ask those who do not upload why they don't upload, and see if we can fix it. 2. Ask those who do not want to upload why they don't provide a download URL. 3. Making it easier to mirror/replicate all metadata. 4. It should be easy to list the files of previous "hidden" versions for a package. 5. Have a --dist-file command for Distribute. 6. Better documentation. Is there anything I missed? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From smueller at cpan.org Sat Dec 26 21:59:40 2009 From: smueller at cpan.org (Steffen Mueller) Date: Sat, 26 Dec 2009 20:59:40 +0000 (UTC) Subject: [Distutils] =?utf-8?q?Python_people_want_CPAN_and_how_the_latter_?= =?utf-8?q?came=09about?= References: <022cb39a9bc238221ab4745be167d2bb@preisshare.net> <319e029f0912210213i7b7bd11bp7f209676977c41ca@mail.gmail.com> <20091221114247.GB23017@rivendell.internal> <319e029f0912211116t134b7e98nd8b24caa37532685@mail.gmail.com> <319e029f0912221015l4ca4ef4dyb0dc13acac6d268a@mail.gmail.com> <319e029f0912241514y1e818590xf8cb9d52dd9ed477@mail.gmail.com> <4B35F1D9.4010400@v.loewis.de> Message-ID: Hi Martin, Martin v. L?wis v.loewis.de> writes: > Maybe it's a terminology matter. "I" have to "authorize": who is "I"? > In PyPI, no person ever authorizes access. Yet, you still cannot upload > newer versions of popular packages. > > Package names are registered on a first-come first-served basis. You > could register a popular package if you were the first one to do so. > However, all the popular packages are already registered, and then only > the person originally performing the registration may upload newer > versions (strictly speaking, that person can also delegate that > permission to others, but that is besides the point that the archive > operators will never need to authorize anything). You're right. It's terminology. This is exactly how PAUSE works. I had a longer reply to Lennart's post written up when firefox bombed out. I should have subscribed to the list instead :/ > > Now, on CPAN, I *can* upload anything even if not authorized to do so. It > > just won't be part of the official indexes if I upload a new version of the > > database interface DBI. > > In PyPI, you can upload the popular package under a different name. It > will be indexed and all, but people still may not use it because of the > different name. Hmm. If you replace "different name" here with "different package name(s)", then it's the same for CPAN. Simply renaming the distribution, however, doesn't work there. > > That we do out meta data stuff on package/namespace/class names as opposed to > > distribution names has the huge benefit of interoperability between > > distributions. > > I think you misunderstood the Python term "distribution" here (or I > misunderstand the point you make). A "distribution" is a tar file (or > such); it's what library authors distribute. There can't be > "interoperability" between them, at least not in the way I understand > "interoperability". Maybe I wasn't clear. But in the end, I think the misunderstanding comes down to a difference between Perl and Python: Perl mixes class names and namespaces (=> class hierarchies instead of namespaces as a language feature) whereas Python has them separate. By distribution, I also meant tarballs. Interoperability in the sense of using library A, B, and C all in the same project (be it library D or an application). If you do that, you need to make sure the fully resolved class names (including their namespace in the Python case) is unique between those libraries. Otherwise there'll be a clash. > I think the point here was that "we" see the advantage that CPAN imposes > with the namespace registrations primarily as theoretical. Yes, it does > prevent two people putting code into Config::Parser, and yes, in theory, > it may be that they do the same in Python with PyPI. In practice, that > is never a problem - there are so many names to chose from, and if you > do happen to conflict with somebody else's naming choice, AND there is > indeed interest in using both packages simultaneously, your users will > ask you to rename your package. However, that happens so rarely if at > all that it hasn't been a real concern. Nit: Renaming packages is really only possible if they have no users. Also, all authors think they wrote the ONE TRUE config parser. So they must have Config::Parser. I humbly think on this point, IF Python namespaces don't do the disambiguation for you anyway, PyPI gets it wrong. On the other hand, if you think in monolithic, large libraries as opposed to small, highly specialized and reusable components that make for the bulk of CPAN, this may not be immediately obvious. You say "namespace registration". I'm not sure what you're refering to, but 99% of CPAN is just the regular first-come auto-registration. The manual registration that is still possible is a mere relic. > If you have malicious users, and unsuspecting users download and run > their code, no namespacing mechanism can stop them, neither in CPAN nor > in PyPI. Malicious users would be able to bypass any checks that are > performed, and experienced users, code review etc. needs to discover > that. That's true. But lowering the barrier by making it easy to upload a new package that has the same name as a popular one but a higher version number is silly. But that is academic. Neither CPAN or PyPI make this mistake. > New or stupid users may accidentally create colliding names. However, > in Python, packages aren't called "db", but, say, "psycopg2", "Twisted", > "django", or "zope". Chances are fairly low that new users accidentally > come up with such a name. DB was a pathological example. It's the class name of the perl-internal debugger but lends itself well to different interpretation. Best regards, Steffen From a.cavallo at cavallinux.eu Sat Dec 26 22:45:10 2009 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Sat, 26 Dec 2009 22:45:10 +0100 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <51069813-92D4-4769-9939-1B6B94B7F4D9@gmail.com> References: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <200912261215.nBQCFEYH026480@theraft.openend.se> <51069813-92D4-4769-9939-1B6B94B7F4D9@gmail.com> Message-ID: <1B3097C2-1112-457C-9D48-5DC1AF3B1AF8@cavallinux.eu> > The PyPI discussions seem to be tending toward mixing the window dressing with the framing, to use a building analogy, and what that will result in is a weak frame and ugly windows. A building that slowly (or quickly) falls down under its own weight, and looks bad doing it. > > I think that splitting > > > package storage and pointers to off-repository storage (for those who don't upload to PyPI) > > metadata about the stored packages > > tools for creating stored packages > > tools for retrieving stored packages > > tools for installing packages > > would go a long way towards unobfuscating this whole discussion Is not what sourceforge already provide? Susebuild server could provide support to complete the loop: > > package storage and pointers to off-repository storage (for those who don't upload to PyPI) > > metadata about the stored packages 1 sorce/project management/metadata (sourceforge) > > tools for creating stored packages > > tools for retrieving stored packages 2 continuous build binary (susebuild) 3 package repositpry download (with programmatic api) on susebuild > > tools for installing packages yum/yast/apt etc tools are already there for linux o and supported on opensuse build server Regards, Antonio If you can forgive my self promotion I've used this approach in a project of mine (pyvm.sf.net): is not completely automated, but it fits the bill point by point: moreover it has support for unitesting too. > > Yes, I'm sure someone will disagree with some fine-point of that division but isn't that what woodshedding is all about? -------------- next part -------------- An HTML attachment was scrubbed... URL: From waterbug at pangalactic.us Sat Dec 26 22:17:04 2009 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sat, 26 Dec 2009 16:17:04 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <51069813-92D4-4769-9939-1B6B94B7F4D9@gmail.com> References: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <200912261215.nBQCFEYH026480@theraft.openend.se> <51069813-92D4-4769-9939-1B6B94B7F4D9@gmail.com> Message-ID: <4B367D50.7040102@pangalactic.us> ssteinerX at gmail.com wrote: > I think that splitting > > > package storage and pointers to off-repository storage (for those who don't upload to PyPI) > > metadata about the stored packages > > tools for creating stored packages > > tools for retrieving stored packages > > tools for installing packages > > would go a long way towards unobfuscating this whole discussion. > > Yes, I'm sure someone will disagree with some fine-point of that > division but isn't that what woodshedding is all about? OT (diction): Hmm ... I think you meant "bikeshedding" (energetic discussion of meaningless details) -- "woodshedding" *is* an actual slang term, but one that originated among musicians, meaning "to practice one's skills alone" (i.e., to go off to a woodshed where no one can hear you). Steve From ssteinerx at gmail.com Sun Dec 27 00:44:47 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sat, 26 Dec 2009 18:44:47 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B367D50.7040102@pangalactic.us> References: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <200912261215.nBQCFEYH026480@theraft.openend.se> <51069813-92D4-4769-9939-1B6B94B7F4D9@gmail.com> <4B367D50.7040102@pangalactic.us> Message-ID: <144D4FE8-3C23-4749-A67C-DB630F693E00@gmail.com> On Dec 26, 2009, at 4:17 PM, Stephen Waterbury wrote: > ssteinerX at gmail.com wrote: >> I think that splitting > package storage and pointers to off-repository storage (for those who don't upload to PyPI) >> > metadata about the stored packages >> > tools for creating stored packages >> > tools for retrieving stored packages >> > tools for installing packages >> would go a long way towards unobfuscating this whole discussion. >> Yes, I'm sure someone will disagree with some fine-point of that >> division but isn't that what woodshedding is all about? > > OT (diction): > Hmm ... I think you meant "bikeshedding" (energetic discussion of > meaningless details) -- "woodshedding" *is* an actual slang term, but > one that originated among musicians, meaning "to practice one's skills > alone" (i.e., to go off to a woodshed where no one can hear you). I know that, I'm a musician, too..it was kind of a play on the fact that simple bikeshedding seems to turn into woodshedding i.e. it goes beyond the normal bikeshedding and becomes beating a dead bikeshed... I couldn't come up with a conjoined word fast enough so I just let it go as I wrote it. S From lregebro at jarn.com Sun Dec 27 09:27:22 2009 From: lregebro at jarn.com (Lennart Regebro) Date: Sun, 27 Dec 2009 09:27:22 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> Message-ID: <319e029f0912270027q6f72b356le65271b8a37a4432@mail.gmail.com> On Sat, Dec 26, 2009 at 18:40, Lennart Regebro wrote: > 3. Making it easier to mirror/replicate all metadata. I've now written a script that does this via XML-RPC. It's dead easy, actually. It's however also slow, and looks like it takes at least five hours. But that's to download all metadata, and there is quite a lot of it. :) Syncing after the initial download is reasonably fast. I wonder if the initial slow download is a problem? It certainly makes it harder to run tests/queries on the complete dataset if you need to wait a day before you can do it. But if you want to set up a third-party service it shouldn't be great hindrance. It takes way longer than that to develop the service. :) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From ssteinerx at gmail.com Sun Dec 27 11:30:22 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 27 Dec 2009 05:30:22 -0500 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912270027q6f72b356le65271b8a37a4432@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270027q6f72b356le65271b8a37a4432@mail.gmail.com> Message-ID: <511FA192-3344-4329-9C08-8DC415038442@gmail.com> On Dec 27, 2009, at 3:27 AM, Lennart Regebro wrote: > On Sat, Dec 26, 2009 at 18:40, Lennart Regebro wrote: >> 3. Making it easier to mirror/replicate all metadata. > > I've now written a script that does this via XML-RPC. It's dead easy, > actually. It's however also slow, and looks like it takes at least > five hours. But that's to download all metadata, and there is quite a > lot of it. :) Syncing after the initial download is reasonably fast. How hard would it be to set up a cron to tar up a daily snapshot so that the initial download was quick (no API calls), then you'd only need an update from the last snapshot? Thanks, S From lregebro at jarn.com Sun Dec 27 11:40:55 2009 From: lregebro at jarn.com (Lennart Regebro) Date: Sun, 27 Dec 2009 11:40:55 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <511FA192-3344-4329-9C08-8DC415038442@gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270027q6f72b356le65271b8a37a4432@mail.gmail.com> <511FA192-3344-4329-9C08-8DC415038442@gmail.com> Message-ID: <319e029f0912270240o438c6665qf32f164b886346c0@mail.gmail.com> On Sun, Dec 27, 2009 at 11:30, ssteinerX at gmail.com wrote: > How hard would it be to set up a cron to tar up a daily snapshot so that the initial download was quick (no API calls), then you'd only need an update from the last snapshot? Not hard, I think. I haven't done a complete download, but the script I made simply read in all the data in memory and then dumped the huge dictionary to a pickle. Updates would be done by loading the dictionary into memory again, and then updating that dictionary with whatever happened since the last time, and dumping it to a pickle again. :) The biggest problem with that technique is the memory usage, but I didn't see it go up significantly during my one-hour test run, so I think it is in fact feasible. If not I guess each package could be dumped into the pickle separately, but that would make updating more complicated. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From lregebro at jarn.com Sun Dec 27 11:47:05 2009 From: lregebro at jarn.com (Lennart Regebro) Date: Sun, 27 Dec 2009 11:47:05 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> Message-ID: <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> On Sat, Dec 26, 2009 at 18:40, Lennart Regebro wrote: > 1. Ask those who do not upload why they don't upload, and see if we can fix it. > 2. Ask those who do not want to upload why they don't provide a download URL. Out of a total of 8522 packages on PyPI, there are 203 packages (2.4%) whose latest release does not provide either a package on PyPI, nor a download url. Of these 16 does not provide any contact data. There are, as far as I can figure out, around 150 individuals to contact. That's enough people that a questionnaire might be useful, with answer that we guess are going to be the common ones. Like: Why did you not upload to PyPI, you bum? 1. I didn't know you could upload to PyPI. 2. I don't want anyone but my servers to have the downloads, thank you. [Why?] 3. My companies policy does not allow it. [Why?] 4. The distutils "sdist upload" procedure doesn't work for me. [Why?] 5. Other. [What?] But there is a download URL metadata. You could have at least filled that in, lazy person! 1. I didn't know you could. 2. Other [What?] Is there significant interest in doing this? In that case, what answer options should we have? -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From martin at v.loewis.de Sun Dec 27 12:16:45 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 27 Dec 2009 12:16:45 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> Message-ID: <4B37421D.5040604@v.loewis.de> > There are, as far as I can figure out, around 150 individuals to > contact. That's enough people that a questionnaire might be useful, > with answer that we guess are going to be the common ones. Like: > > Why did you not upload to PyPI, you bum? > 1. I didn't know you could upload to PyPI. > 2. I don't want anyone but my servers to have the downloads, thank you. [Why?] > 3. My companies policy does not allow it. [Why?] > 4. The distutils "sdist upload" procedure doesn't work for me. [Why?] > 5. Other. [What?] > > But there is a download URL metadata. You could have at least filled > that in, lazy person! > 1. I didn't know you could. > 2. Other [What?] One answer option should be "there is nothing I can release at this point". Regards, Martin From ssteinerx at gmail.com Sun Dec 27 16:54:25 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 27 Dec 2009 10:54:25 -0500 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> Message-ID: <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> On Dec 27, 2009, at 5:47 AM, Lennart Regebro wrote: > On Sat, Dec 26, 2009 at 18:40, Lennart Regebro wrote: >> 1. Ask those who do not upload why they don't upload, and see if we can fix it. >> 2. Ask those who do not want to upload why they don't provide a download URL. > > Out of a total of 8522 packages on PyPI, there are 203 packages (2.4%) > whose latest release does not provide either a package on PyPI, nor a > download url. Of these 16 does not provide any contact data. I'll update the PyPI checker in the distutilsversion test_pypi_versions.py to provide that information as well. I think I'll make a separate project (uploaded to PyPI, promise) out of it so we have a shared way to get statistics on PyPI projects. Is your code for this in the PyPI code repository or were these just quick one-off's? If there's no data on PyPI and no download url then wouldn't those be "non-packages?" And if there's no contact info, "non-package" by "nobody?" Sounds like a song title. The survey should include a "Project abandoned" or "Nothing to upload" or "It was just a mistake" and an "Ok to delete immediately". Any non-project where the owner gives permission to delete, where the e-mail bounces, or there's no response in a month, just delete it. That 2.4% is pretty small compared to somewhere like SourceForge where every other project seems to be abandoned or worse and there sure are a lot of them... S From lregebro at jarn.com Sun Dec 27 17:27:03 2009 From: lregebro at jarn.com (Lennart Regebro) Date: Sun, 27 Dec 2009 17:27:03 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> Message-ID: <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> On Sun, Dec 27, 2009 at 16:54, ssteinerX at gmail.com wrote: > I'll update the PyPI checker in the distutilsversion test_pypi_versions.py to provide that information as well. ?I think I'll make a separate project (uploaded to PyPI, promise) out of it so we have a shared way to get statistics on PyPI projects. ?Is your code for this in the PyPI code repository or were these just quick one-off's? Quick one-offs. Here, in fact: Takes an hour or two to run. from xmlrpc import client PYPIURL = 'http://pypi.python.org/pypi' pypi = client.Server(PYPIURL) for package in pypi.list_packages(): for release in pypi.package_releases(package, False): release_urls = pypi.release_urls(package, release) if release_urls: continue release_data = pypi.release_data(package, release) if not release_data.get('download_url', ''): print("Package %s, Release: %s" % (package, release)) print(" Did not have releases on PyPI or download URL") print(" Author: %s <%s>" % (release_data['author'], release_data['author_email'])) print(" Maintainer: %s <%s>" % (release_data['maintainer'], release_data['maintainer_email'])) > If there's no data on PyPI and no download url then wouldn't those be "non-packages?" ? And if there's no contact info, "non-package" by "nobody?" ?Sounds like a song title. :) > The survey should include a "Project abandoned" or "Nothing to upload" or "It was just a mistake" and an "Ok to delete immediately". Right. > Any non-project where the owner gives permission to delete, where the e-mail bounces, or there's no response in a month, just delete it. There may be more people who have the rights to the system, so in fact in these cases we should check who has Owner rights and contact all of them. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 From ssteinerx at gmail.com Sun Dec 27 17:50:30 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 27 Dec 2009 11:50:30 -0500 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> Message-ID: On Dec 27, 2009, at 11:27 AM, Lennart Regebro wrote: > On Sun, Dec 27, 2009 at 16:54, ssteinerX at gmail.com wrote: >> I'll update the PyPI checker in the distutilsversion test_pypi_versions.py to provide that information as well. I think I'll make a separate project (uploaded to PyPI, promise) out of it so we have a shared way to get statistics on PyPI projects. Is your code for this in the PyPI code repository or were these just quick one-off's? > > Quick one-offs. Here, in fact: > > Takes an hour or two to run. > > from xmlrpc import client > PYPIURL = 'http://pypi.python.org/pypi' > pypi = client.Server(PYPIURL) > for package in pypi.list_packages(): > for release in pypi.package_releases(package, False): > release_urls = pypi.release_urls(package, release) > if release_urls: > continue > release_data = pypi.release_data(package, release) > if not release_data.get('download_url', ''): > print("Package %s, Release: %s" % (package, release)) > print(" Did not have releases on PyPI or download URL") > print(" Author: %s <%s>" % (release_data['author'], > release_data['author_email'])) > print(" Maintainer: %s <%s>" % (release_data['maintainer'], > > release_data['maintainer_email'])) Ok, thanks, I'll throw that into the code, in some form. The tarred download would be really handy for this utility as, if there's no .pkl of the data, or the user requests it, I pull fresh copy. Right now, my query is very limited (I'm only looking for version info) and only takes a couple of minutes to build. Since I'm going to add more capabilities, having a quick way to refresh the whole thing would be great. I'll put my version up in the new project and maybe we can work together to get it into some PyPI code or to store the version I build somewhere though building it right on the server would seem to be much faster (if memory intensive). Thanks! S aka/Steve Steiner aka/ssteinerX From martin at v.loewis.de Sun Dec 27 19:34:17 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 27 Dec 2009 19:34:17 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> Message-ID: <4B37A8A9.4080908@v.loewis.de> > The tarred download would be really handy for this utility as, if > there's no .pkl of the data, or the user requests it, I pull fresh > copy. How difficult would it be for you to provide such data, for anybody interested in using them? Regards, Martin From lregebro at jarn.com Sun Dec 27 20:10:53 2009 From: lregebro at jarn.com (Lennart Regebro) Date: Sun, 27 Dec 2009 20:10:53 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> Message-ID: <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> On Sun, Dec 27, 2009 at 17:50, ssteinerX at gmail.com wrote: > The tarred download would be really handy for this utility as, if there's no .pkl of the data, or the user requests it, I pull ?fresh copy. Right... Anyone want to host such a tarred download? :-) I pretty much got the code, but don't feel like setting up and hosting it right now. Maybe later. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ssteinerx at gmail.com Sun Dec 27 20:31:18 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 27 Dec 2009 14:31:18 -0500 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <4B37A8A9.4080908@v.loewis.de> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> <4B37A8A9.4080908@v.loewis.de> Message-ID: <1D980FA2-B983-481F-B331-096684A9A9B8@gmail.com> On Dec 27, 2009, at 1:34 PM, Martin v. L?wis wrote: >> The tarred download would be really handy for this utility as, if >> there's no .pkl of the data, or the user requests it, I pull fresh >> copy. > > How difficult would it be for you to provide such data, for anybody > interested in using them? Really easy, that's what I'm saying, let's cooperate in getting this put somewhere. I have plenty of hosting resources and can put a cron job up to keep it up to date. S From ssteinerx at gmail.com Sun Dec 27 20:32:36 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 27 Dec 2009 14:32:36 -0500 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> Message-ID: <74F01CDD-3D42-48F4-B1EA-71D13B5B4E81@gmail.com> On Dec 27, 2009, at 2:10 PM, Lennart Regebro wrote: > On Sun, Dec 27, 2009 at 17:50, ssteinerX at gmail.com wrote: >> The tarred download would be really handy for this utility as, if there's no .pkl of the data, or the user requests it, I pull fresh copy. > > Right... Anyone want to host such a tarred download? :-) > I pretty much got the code, but don't feel like setting up and hosting > it right now. Maybe later. I'll finish up the code, get it set up on a cron job to keep it up to date, and host it. I'll probably just put the finished data up on S3, no big deal and I can run the cron job on one of our servers. S From ssteinerx at gmail.com Sun Dec 27 23:07:51 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 27 Dec 2009 17:07:51 -0500 Subject: [Distutils] PyPI Data Synchronizer In-Reply-To: <319e029f0912271134y56be4354hea037e0e0ffb8c24@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> <74F01CDD-3D42-48F4-B1EA-71D13B5B4E81@gmail.com> <319e029f0912271134y56be4354hea037e0e0ffb8c24@mail.gmail.com> Message-ID: <8C020BBB-BB45-409B-BF11-8F67099AE8B3@gmail.com> On Dec 27, 2009, at 2:34 PM, Lennart Regebro wrote in Re: [Distutils] How Python can have CPAN.: > I attached the file I used for the mirroring. Cool. I'll just integrate it with the current version of the PyPI version extractor from the distutilsversion project and upload it to PyPI as a new project. I'll make sure to put up a usable client class that pulls from the stored version and updates it, with appropriate caching similar to what you did in your code (and the way that I did it in my version as well). That way, anyone wishing to write a tool has a drop-in class that will use minimal resources to get a full copy of the PyPI data. I'll put a copy on one of our servers, set a sync job to run every day then push the result to S3. I'll publish the URL on this list (once, it won't change). Feel free to put a link to it from the PyPI site, and anyone needing the data can just pull it from there, presumably using the supplied class. I'll probably start this tonight, I'll post it as soon as it's working. S From martin at v.loewis.de Sun Dec 27 22:49:28 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 27 Dec 2009 22:49:28 +0100 Subject: [Distutils] PyPI Data Synchronizer In-Reply-To: <8C020BBB-BB45-409B-BF11-8F67099AE8B3@gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> <74F01CDD-3D42-48F4-B1EA-71D13B5B4E81@gmail.com> <319e029f0912271134y56be4354hea037e0e0ffb8c24@mail.gmail.com> <8C020BBB-BB45-409B-BF11-8F67099AE8B3@gmail.com> Message-ID: <4B37D668.5020803@v.loewis.de> > I'll make sure to put up a usable client class that pulls from the > stored version and updates it, with appropriate caching similar to > what you did in your code (and the way that I did it in my version as > well). Please make sure to change the User-Agent header in the requests, so that we can attribute accesses to this tool. Regards, Martin From ssteinerx at gmail.com Mon Dec 28 00:02:37 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 27 Dec 2009 18:02:37 -0500 Subject: [Distutils] PyPI Data Synchronizer In-Reply-To: <4B37D668.5020803@v.loewis.de> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> <74F01CDD-3D42-48F4-B1EA-71D13B5B4E81@gmail.com> <319e029f0912271134y56be4354hea037e0e0ffb8c24@mail.gmail.com> <8C020BBB-BB45-409B-BF11-8F67099AE8B3@gmail.com> <4B37D668.5020803@v.loewis.de> Message-ID: <0E1E2DE8-B00F-4007-9261-54686D914A5B@gmail.com> On Dec 27, 2009, at 4:49 PM, Martin v. L?wis wrote: >> I'll make sure to put up a usable client class that pulls from the >> stored version and updates it, with appropriate caching similar to >> what you did in your code (and the way that I did it in my version as >> well). > > Please make sure to change the User-Agent header in the requests, so > that we can attribute accesses to this tool. Ok, I'll identify the tool and my cron'd collector specifically so we can track resource usage for the tool in general and specifically for my collector, just in case it becomes an issue down the road. Steve aka/S aka/ssteinerX From david.lyon at preisshare.net Mon Dec 28 00:24:27 2009 From: david.lyon at preisshare.net (david.lyon at preisshare.net) Date: Mon, 28 Dec 2009 10:24:27 +1100 (EST) Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4B367D50.7040102@pangalactic.us> References: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <200912261215.nBQCFEYH026480@theraft.openend.se> <51069813-92D4-4769-9939-1B6B94B7F4D9@gmail.com> <4B367D50.7040102@pangalactic.us> Message-ID: <4106.115.128.50.49.1261956267.squirrel@syd-srv02.ezyreg.com> > ssteinerX at gmail.com wrote: > Hmm ... I think you meant "bikeshedding" (energetic discussion of > meaningless details) -- "woodshedding" *is* an actual slang term, but > one that originated among musicians, meaning "to practice one's skills > alone" (i.e., to go off to a woodshed where no one can hear you). Sometimes a woodshed in the country is the best place to write music. There's a special character that you get in the music that isn't possible living in the city. Country air.. David From david.lyon at preisshare.net Mon Dec 28 00:42:14 2009 From: david.lyon at preisshare.net (david.lyon at preisshare.net) Date: Mon, 28 Dec 2009 10:42:14 +1100 (EST) Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> Message-ID: <4180.115.128.50.49.1261957334.squirrel@syd-srv02.ezyreg.com> > On Sun, Dec 27, 2009 at 17:50, ssteinerX at gmail.com > wrote: >> The tarred download would be really handy for this utility as, if >> there's no .pkl of the data, or the user requests it, I pull ?fresh >> copy. > > Right... Anyone want to host such a tarred download? :-) > I pretty much got the code, but don't feel like setting up and hosting > it right now. Maybe later. I have some spare on a custom domain. What do you need ? David From ziade.tarek at gmail.com Mon Dec 28 00:51:59 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 28 Dec 2009 00:51:59 +0100 Subject: [Distutils] Finishing PEP 345 In-Reply-To: <4B353C81.1080702@v.loewis.de> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> <4B3279E9.4010507@v.loewis.de> <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> <4B3324B2.3030708@v.loewis.de> <94bdd2610912240132t345b4d8bnc1bc81d5ae82b194@mail.gmail.com> <4B334626.6040505@v.loewis.de> <94bdd2610912250412m4402a238j741ce6c3bc7ba53d@mail.gmail.com> <4B353C81.1080702@v.loewis.de> Message-ID: <94bdd2610912271551u4a92ffbbr5b3496f4c07af822@mail.gmail.com> I've updated PEP 345 with this feedback. I've made the range (~=) operator the default one, and added more details on how versions works. Regards Tarek From tseaver at palladion.com Mon Dec 28 00:52:32 2009 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 27 Dec 2009 18:52:32 -0500 Subject: [Distutils] Python people want CPAN and how the latter came about In-Reply-To: <4106.115.128.50.49.1261956267.squirrel@syd-srv02.ezyreg.com> References: <5b8d13220912242322h46602724x6a86dcb500259aa@mail.gmail.com> <319e029f0912250052o321992bg5393ed47c0b21800@mail.gmail.com> <5b8d13220912250106g4cc44036r6ae1cca4fd1706f9@mail.gmail.com> <319e029f0912250112v44b3328cv7d9c88831dc6659b@mail.gmail.com> <5b8d13220912250133tb371158q6d2a6e2883606e69@mail.gmail.com> <319e029f0912250155q6392232dgda9b0b6c743b5137@mail.gmail.com> <5b8d13220912250243w3e0a6734qf49114e63873adf5@mail.gmail.com> <200912251122.nBPBM5pP004260@theraft.openend.se> <5b8d13220912252003w5b220b9bq3bd1b1312c2d258e@mail.gmail.com> <200912261215.nBQCFEYH026480@theraft.openend.se> <51069813-92D4-4769-9939-1B6B94B7F4D9@gmail.com> <4B367D50.7040102@pangalactic.us> <4106.115.128.50.49.1261956267.squirrel@syd-srv02.ezyreg.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 david.lyon at preisshare.net wrote: >> ssteinerX at gmail.com wrote: >> Hmm ... I think you meant "bikeshedding" (energetic discussion of >> meaningless details) -- "woodshedding" *is* an actual slang term, but >> one that originated among musicians, meaning "to practice one's skills >> alone" (i.e., to go off to a woodshed where no one can hear you). > > Sometimes a woodshed in the country is the best place to write music. > There's a special character that you get in the music that isn't possible > living in the city. On another face of living in the country: the woodshed is also where recalcitrant youth receive their more-or-less-well-earned beatings. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks38zYACgkQ+gerLs4ltQ5EqgCdGFdBkmY40ohDy8zFibsTlD3v jjsAn3KOf4tGCvh1dWQM1eZ05zZty19z =3Maz -----END PGP SIGNATURE----- From ssteinerx at gmail.com Mon Dec 28 02:25:02 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 27 Dec 2009 20:25:02 -0500 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <4180.115.128.50.49.1261957334.squirrel@syd-srv02.ezyreg.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <885FBB40-0932-423C-9E3D-F5DA7318F136@gmail.com> <319e029f0912270827r6107dde8wbba9913fa31b982d@mail.gmail.com> <319e029f0912271110v44ec2e45k4f113afa73a4bf22@mail.gmail.com> <4180.115.128.50.49.1261957334.squirrel@syd-srv02.ezyreg.com> Message-ID: On Dec 27, 2009, at 6:42 PM, david.lyon at preisshare.net wrote: >> On Sun, Dec 27, 2009 at 17:50, ssteinerX at gmail.com >> wrote: >>> The tarred download would be really handy for this utility as, if >>> there's no .pkl of the data, or the user requests it, I pull fresh >>> copy. >> >> Right... Anyone want to host such a tarred download? :-) >> I pretty much got the code, but don't feel like setting up and hosting >> it right now. Maybe later. > > I have some spare on a custom domain. What do you need ? I have it covered. S From david.lyon at preisshare.net Wed Dec 30 05:29:35 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 29 Dec 2009 23:29:35 -0500 Subject: [Distutils] Finishing PEP 345 - Code-repository field In-Reply-To: <94bdd2610912250412m4402a238j741ce6c3bc7ba53d@mail.gmail.com> References: <94bdd2610912180641i6df1a26co9c3ff34aba034982@mail.gmail.com> <4B3224B5.8000503@v.loewis.de> <94bdd2610912231045h5df841f5pe90cd46b387b0755@mail.gmail.com> <4B3266CF.7030705@v.loewis.de> <94bdd2610912231123o6d3236c7j289df78391083ac5@mail.gmail.com> <4B3279E9.4010507@v.loewis.de> <94bdd2610912231311h44e5563bn98fab09cab3c37f1@mail.gmail.com> <4B3324B2.3030708@v.loewis.de> <94bdd2610912240132t345b4d8bnc1bc81d5ae82b194@mail.gmail.com> <4B334626.6040505@v.loewis.de> <94bdd2610912250412m4402a238j741ce6c3bc7ba53d@mail.gmail.com> Message-ID: <1cbd7c543edadc9c0ee0c7e50f26e1b0@preisshare.net> Hi Tarek, There's one final field that I think that we need and that's to be able for a software releaser to specify a code-repository feed source. It's different than a download url, which may point to a software 'release'. The code repository is a changing feed source for software updates and is most likely to be cvs, svn, bzr or hg based link. In this way, it makes it possible to have package updating in place to keep python code up to date, if that is the users wish. Of course, it is up to the user to install whatever SCM system that is necessary for the update function. The value of the code-repository field would be prefixed with cvs:, svn:, bzr: or hg: or whatever SCM prefix is used. It should never be allowed to be a http: or ftp: link. What this allows a user to do, is more easily install code from the various software repositories that we find dotted all over the internet these days. They're more fluid than 'download' links and better suited to the distribution of 'code', as in python code. David From setuptools at bugs.python.org Wed Dec 30 12:41:33 2009 From: setuptools at bugs.python.org (Dennis) Date: Wed, 30 Dec 2009 11:41:33 +0000 Subject: [Distutils] [issue100] setuptools and easy-install.pth In-Reply-To: <1262173293.15.0.600457939371.issue100@psf.upfronthosting.co.za> Message-ID: <1262173293.15.0.600457939371.issue100@psf.upfronthosting.co.za> New submission from Dennis : Caveat: I am not wise in the ways of Pythonology and may misunderstand the use of easy-install.pth, so please bear with me. What started this off was an attempt to install the latest calibre. It complained it could not find lxml (even though it was installed). So I recompiled lxml (just to be sure) then calibre; it then complained it could not find mechanize. So I reinstalled mechanize (it was already installed) and then calibre complains it cannot find lxml; rinse and repeat. In the end I just manually added the eggs needed by calibre to easy-install.pth and it compiled fine. Then just to see what happens I recompiled setuptools and it blew away everything I had added to easy-install.pth leaving just the setuptools egg. So I started looking at setuptools and installing from source is straight forward; python setup.py build && python setup.py install and end up with; /usr/lib/python2.6/site-packages/easy-install.pth /usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg /usr/lib/python2.6/site-packages/setuptools.pth and easy-install.pth contains; import sys; sys.__plen = len(sys.path) ./setuptools-0.6c11-py2.6.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) Now if I install lxml, easy-install.pth contains; import sys; sys.__plen = len(sys.path) ./lxml-2.2.4-py2.6-linux-x86_64.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) What happened to ./setuptools-0.6c11-py2.6.egg? So I recompile setuptools and now easy-install.pth has removed the lxml egg and I end up where I started. So setuptools over writes easy-install.pth instead of adding itself. This happens with a number of other python apps that use setuptools/easy-install.pth. In the docs I saw a mention of running setuptools-0.6c11-py2.6.egg as a script. Had no problems doing that, except its behavior is the same as the other build process. At this point I am a little confused about whether easy-install.pth should be appended by setuptools or actually over written. If it is the latter, it seems counter-intuitive. ---------- messages: 483 nosy: dveatch priority: bug status: unread title: setuptools and easy-install.pth _______________________________________________ Setuptools tracker _______________________________________________ From sdouche at gmail.com Wed Dec 30 19:48:34 2009 From: sdouche at gmail.com (Sebastien Douche) Date: Wed, 30 Dec 2009 19:48:34 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> Message-ID: <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> On Sun, Dec 27, 2009 at 11:47, Lennart Regebro wrote: > Out of a total of 8522 packages on PyPI, there are 203 packages (2.4%) > whose latest release does not provide either a package on PyPI, nor a > download url. Of these 16 does not provide any contact data. Hi Lennart, Glad to see someone is interested by a PyPI mirror, I have one here and it's a pity. Statistics (from the creation of the mirror / proxy. The goal is to avoid external download, like an internal debian mirror): 2009-12-15 21:37:20,855 DEBUG Found (cached): 0 2009-12-15 21:37:20,855 DEBUG Stored (downloaded): 15367 2009-12-15 21:37:20,855 DEBUG Not found (404): 188 2009-12-15 21:37:20,855 DEBUG Invalid packages: 0 2009-12-15 21:37:20,855 DEBUG Invalid URLs: 54 2009-12-15 21:37:20,855 DEBUG Runtime: 208m38s The root issue (for me) is: packages out of the PyPI. A lot of broken links, broken html pages or stupid scripts (cf. old SourceForge). Some examples: WARNING Unload downloading http://wiki.woodpecker.org.cn/moin/UliPad (timed out) WARNING Unload downloading http://launchpad.net/mcrepogen/+download (The read operation timed out) WARNING Unload downloading http://launchpad.net/mcrepogen (The read operation timed out) WARNING Unload downloading https://launchpad.net/lovely.tal (The read operation timed out) WARNING Unload downloading ffnet.sourceforge.net (unknown url type: ffnet.sourceforge.net) WARNING Unload downloading http://pysqlite.org/ ((-3, 'Temporary failure in name resolution')) > Is there significant interest in doing this? YES! ;) In that case, what answer > options should we have? Always upload a version to PyPI, the only way to have a reliable, solid and smart PyPI and an easy way to proxy-ing. Think the case where SF is down: No docutils. Zope server down: no Zope 2, no Zope3, no ZTK, no buildout... With a full mirror I don't care... Note: I'm very happy when I see a distribution with: - a description - a summary (with examples if necessary) - a changelog (quick way to see what's new) - the name of the author and email (or maintainer) - contain files (with distribution name = package name, not MyPackage and mypackage) Like this : http://pypi.python.org/pypi/collective.portlet.relateditems/0.3.0 And not this: http://pypi.python.org/pypi/django-sphinxdoc/0.2 Cheers -- Sebastien Douche Twitter: http://bit.ly/afkrK (agile, python, open source) From ssteinerx at gmail.com Wed Dec 30 19:57:57 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 30 Dec 2009 13:57:57 -0500 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> Message-ID: <3F0F156F-FE3D-4E9D-BC9B-191A03806163@gmail.com> On Dec 30, 2009, at 1:48 PM, Sebastien Douche wrote: > On Sun, Dec 27, 2009 at 11:47, Lennart Regebro wrote: > >> Out of a total of 8522 packages on PyPI, there are 203 packages (2.4%) >> whose latest release does not provide either a package on PyPI, nor a >> download url. Of these 16 does not provide any contact data. > > Hi Lennart, > Glad to see someone is interested by a PyPI mirror, I have one here > and it's a pity. > > Statistics (from the creation of the mirror / proxy. The goal is to > avoid external download, like an internal debian mirror): > 2009-12-15 21:37:20,855 DEBUG Found (cached): 0 > 2009-12-15 21:37:20,855 DEBUG Stored (downloaded): 15367 > 2009-12-15 21:37:20,855 DEBUG Not found (404): 188 > 2009-12-15 21:37:20,855 DEBUG Invalid packages: 0 > 2009-12-15 21:37:20,855 DEBUG Invalid URLs: 54 > 2009-12-15 21:37:20,855 DEBUG Runtime: 208m38s > > The root issue (for me) is: packages out of the PyPI. A lot of broken > links, broken html pages or stupid scripts (cf. old SourceForge). I will put a way of getting this data out, thanks for the heads up. > Some examples: > WARNING Unload downloading http://wiki.woodpecker.org.cn/moin/UliPad (timed out) > WARNING Unload downloading http://launchpad.net/mcrepogen/+download > (The read operation timed out) > WARNING Unload downloading http://launchpad.net/mcrepogen (The read > operation timed out) > WARNING Unload downloading https://launchpad.net/lovely.tal (The read > operation timed out) > WARNING Unload downloading ffnet.sourceforge.net (unknown url type: > ffnet.sourceforge.net) > WARNING Unload downloading http://pysqlite.org/ ((-3, 'Temporary > failure in name resolution')) > > >> Is there significant interest in doing this? > > YES! ;) > > In that case, what answer >> options should we have? > > Always upload a version to PyPI, the only way to have a reliable, > solid and smart PyPI and an easy way to proxy-ing. Think the case > where SF is down: No docutils. Zope server down: no Zope 2, no Zope3, > no ZTK, no buildout... With a full mirror I don't care... > > Note: I'm very happy when I see a distribution with: > - a description > - a summary (with examples if necessary) > - a changelog (quick way to see what's new) > - the name of the author and email (or maintainer) > - contain files (with distribution name = package name, not MyPackage > and mypackage) > > > Like this : > http://pypi.python.org/pypi/collective.portlet.relateditems/0.3.0 > > And not this: > http://pypi.python.org/pypi/django-sphinxdoc/0.2 I have put this into my working spec document which I'll be publishing with the first version of the code (which won't have all the options implemented, but they'll be in the plan/issue tracker in case anyone wants to help). Steve From sdouche at gmail.com Wed Dec 30 19:59:29 2009 From: sdouche at gmail.com (Sebastien Douche) Date: Wed, 30 Dec 2009 19:59:29 +0100 Subject: [Distutils] [distribute] Strategy for a DVCS extension Message-ID: <5e1183fa0912301059k2aea1a07j5acd28dafd36506a@mail.gmail.com> Hi folks :) I want to know the strategy to be followed in a DVCS extension. A small improvement I want to code is unique area for the version (and not version.cfg *and* setup.py) What do you think? Any ideas welcome. -- Sebastien Douche Twitter: http://bit.ly/afkrK (agile, python, open source) From ssteinerx at gmail.com Wed Dec 30 20:38:37 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 30 Dec 2009 14:38:37 -0500 Subject: [Distutils] [distribute] Strategy for a DVCS extension In-Reply-To: <5e1183fa0912301059k2aea1a07j5acd28dafd36506a@mail.gmail.com> References: <5e1183fa0912301059k2aea1a07j5acd28dafd36506a@mail.gmail.com> Message-ID: <43203376-7E43-425B-9473-949D5BF33319@gmail.com> On Dec 30, 2009, at 1:59 PM, Sebastien Douche wrote: > Hi folks :) > I want to know the strategy to be followed in a DVCS extension. A > small improvement I want to code is unique area for the version (and > not version.cfg *and* setup.py) > > What do you think? Any ideas welcome Sorry, I'm not sure what you're proposing or asking. A DVCS extension of what? S From lregebro at jarn.com Wed Dec 30 22:39:46 2009 From: lregebro at jarn.com (Lennart Regebro) Date: Wed, 30 Dec 2009 22:39:46 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> Message-ID: <319e029f0912301339u592da7e9u5fbf6896c07320a6@mail.gmail.com> On Wed, Dec 30, 2009 at 19:48, Sebastien Douche wrote: >> Is there significant interest in doing this? > > YES! ;) > > In that case, what answer >> options should we have? > > Always upload a version to PyPI, the only way to have a reliable, The question was if there was interest in sending out a questionnaire to maintainers. Forcing uploads to PyPI is a debate that has been flogged to death. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From sridharr at activestate.com Wed Dec 30 22:42:25 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 30 Dec 2009 13:42:25 -0800 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <3F0F156F-FE3D-4E9D-BC9B-191A03806163@gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> <3F0F156F-FE3D-4E9D-BC9B-191A03806163@gmail.com> Message-ID: <4B3BC941.8080607@activestate.com> On 12/30/2009 10:57 AM, ssteinerX at gmail.com wrote: > On Dec 30, 2009, at 1:48 PM, Sebastien Douche wrote: > >> > On Sun, Dec 27, 2009 at 11:47, Lennart Regebro wrote: >> > >>> >> Out of a total of 8522 packages on PyPI, there are 203 packages (2.4%) >>> >> whose latest release does not provide either a package on PyPI, nor a >>> >> download url. Of these 16 does not provide any contact data. >> > >> > Hi Lennart, >> > Glad to see someone is interested by a PyPI mirror, I have one here >> > and it's a pity. >> > >> > Statistics (from the creation of the mirror / proxy. The goal is to >> > avoid external download, like an internal debian mirror): >> > 2009-12-15 21:37:20,855 DEBUG Found (cached): 0 >> > 2009-12-15 21:37:20,855 DEBUG Stored (downloaded): 15367 >> > 2009-12-15 21:37:20,855 DEBUG Not found (404): 188 >> > 2009-12-15 21:37:20,855 DEBUG Invalid packages: 0 >> > 2009-12-15 21:37:20,855 DEBUG Invalid URLs: 54 >> > 2009-12-15 21:37:20,855 DEBUG Runtime: 208m38s >> > >> > The root issue (for me) is: packages out of the PyPI. A lot of broken >> > links, broken html pages or stupid scripts (cf. old SourceForge). > I will put a way of getting this data out, thanks for the heads up. Greetings Sebastien and Steve, The way of getting [external packages] was already implemented. It is called `setuptools.package_index` which is what we use in our internal mirror program (planning to open-source and, perhaps also, host it publicly) which also does the metadata extraction (PKG-INFO, requires.txt) and index files that I mentioned earlier. It is of no use to pity z3c.pypimirror or any other mirror program, because the issue is not with those programs, but with the lack of a central archive from which all sources and metadata can be reliably mirrored. I will, once again, draw the reader's attention to the following: [Steffen Mueller] > My thesis is that the huge success of the CPAN has been facilitated by > two factors[2]. The first is simplicity. When Jarkko Hietaniemi > originally came up with it, the CPAN was (and mostly still is) just an > FTP archive with a by-author directory structure that is mirrored many > times. http://www.mail-archive.com/distutils-sig at python.org/msg10537.html -srid From david.lyon at preisshare.net Wed Dec 30 23:15:11 2009 From: david.lyon at preisshare.net (david.lyon at preisshare.net) Date: Thu, 31 Dec 2009 09:15:11 +1100 (EST) Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912301339u592da7e9u5fbf6896c07320a6@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> <319e029f0912301339u592da7e9u5fbf6896c07320a6@mail.gmail.com> Message-ID: <1127.115.128.8.221.1262211311.squirrel@syd-srv02.ezyreg.com> > On Wed, Dec 30, 2009 at 19:48, Sebastien Douche wrote: >>> Is there significant interest in doing this? >> >> YES! ;) >> >> In that case, what answer >>> options should we have? >> >> Always upload a version to PyPI, the only way to have a reliable, > > The question was if there was interest in sending out a questionnaire > to maintainers. > Forcing uploads to PyPI is a debate that has been flogged to death. In this day and age it just may not viable to do that. If PEP-345 could be adjusted to have a code a Code-Repository option then it wouldn't be so difficult to use a bot on pypi to pull code *in*, test it and package it. Developers don't always have time to drop back to a command line and build and upload using a command line tool that takes 30 seconds. Especially already after they have done a 'hg push' or 'svn commit..' to their own repository. I'd hazzard a guess but I'd say that 80% of pypi projects would be better served with a (external) code repository reference than actually keeping everything built on pypi. And asking the package creators to do that. Here, I don't want to throw away pypi. Clearly it needs to stay and retain its traditional operating mode. I'm just making the point that a simpler Metadata based solution could might serve the needs of users more. David From david.lyon at preisshare.net Thu Dec 31 00:11:39 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 30 Dec 2009 18:11:39 -0500 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> Message-ID: On Wed, 30 Dec 2009 19:48:34 +0100, Sebastien Douche wrote: > Glad to see someone is interested by a PyPI mirror, I have one here > and it's a pity. How did you make it? > Note: I'm very happy when I see a distribution with: > - a description > - a summary (with examples if necessary) > - a changelog (quick way to see what's new) > - the name of the author and email (or maintainer) > - contain files (with distribution name = package name, not MyPackage > and mypackage) Sure. > Like this : > http://pypi.python.org/pypi/collective.portlet.relateditems/0.3.0 I just had a quick look at that package, just as an example. Next problem is that it is a python 2.4 Egg.. That is a real problem... what about for *my* whatever version of python x.y. The process of user confusion now starts on what to do and how to get that installed.. We are still not at anything nearing the simplicity of cpan. But it is possible to do something about it - I hope. David From sdouche at gmail.com Thu Dec 31 06:28:20 2009 From: sdouche at gmail.com (Sebastien Douche) Date: Thu, 31 Dec 2009 06:28:20 +0100 Subject: [Distutils] How Python can have CPAN. In-Reply-To: <319e029f0912301339u592da7e9u5fbf6896c07320a6@mail.gmail.com> References: <319e029f0912260940v1466aec6x76f78003e34efbbb@mail.gmail.com> <319e029f0912270247h1d5601edl9d5bcb91092e4b7a@mail.gmail.com> <5e1183fa0912301048w541a4e67ra67b2a93d40ab4a5@mail.gmail.com> <319e029f0912301339u592da7e9u5fbf6896c07320a6@mail.gmail.com> Message-ID: <5e1183fa0912302128x3aa6fac4x12a534b0022be0f2@mail.gmail.com> On Wed, Dec 30, 2009 at 22:39, Lennart Regebro wrote: > The question was if there was interest in sending out a questionnaire > to maintainers. Sorry Lennart. I think it? a good step. Go ahead ;). -- Sebastien Douche Twitter: http://bit.ly/afkrK (agile, python, open source)