>> + ? ? ?<title tal:content="string:${data/title} : Python Package Index" /> > > Seems like a good idea. I've double-checked that the easy_install > "API" doesn't depend on this HTML. Done. Have you seen the second patch? It should be more cool. I've reattached it. > Can we stop the rather pointless arguing over templating engines now? Sure. Pointless arguing is a waste of time. I hoped to get an answer about superior features of TAL, which didn't seem as popular as Django/Jinja for me. Now I see that majority of PyPI devs are more familiar with TAL. So far my biggest concern is template readability. It will help if my editor had syntax highlighting for this XML language. All I need for that is XSD schema. Unfortunately, references contain only EBNF notation. -- anatoly t. -------------- next part -------------- Index: pypi/templates/display.pt =================================================================== --- pypi/templates/display.pt (revision 925) +++ pypi/templates/display.pt (working copy) @@ -4,6 +4,8 @@ xmlns:metal="http://xml.zope.org/namespaces/metal" metal:use-macro="standard_template/macros/page"> +<metal:fill fill-slot="title" tal:content="string:${data/name} ${data/version}: Python Package Index" /> + <metal:fill fill-slot="head"> <meta tal:condition="data/release/keywords | nothing" name="keywords" Index: pypi/templates/standard_template.pt =================================================================== --- pypi/templates/standard_template.pt (revision 925) +++ pypi/templates/standard_template.pt (working copy) @@ -8,7 +8,7 @@ <META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW" tal:condition="data/norobots"/> <meta content="text/html; charset=utf-8" http-equiv="content-type" /> <base tal:attributes="href data/FULL_PATH_INFO"/> - <title tal:content="string:Python Package Index : ${data/title}" /> + <title metal:define-slot="title" tal:content="Python Package Index : string:${data/title}" /> <meta tal:attributes="content data/keywords" /> <meta tal:attributes="content data/description" /> <link rel="alternate" type="application/rss+xml" title="RSS: 30 latest updates" href="http://www.python.org/pypi?:action=rss"/> Index: pypi/webui.py =================================================================== --- pypi/webui.py (revision 925) +++ pypi/webui.py (working copy) @@ -1348,7 +1348,6 @@ name=name, version=version, release=release, description=release.get('summary') or name, keywords=release.get('keywords', ''), - title=name + " " +version, requires=values('requires'), provides=values('provides'), obsoletes=values('obsoletes'), From richard at python.org Fri Jul 1 10:35:27 2011 From: richard at python.org (Richard Jones) Date: Fri, 1 Jul 2011 18:35:27 +1000 Subject: [Catalog-sig] Tab Title customization and Tal to Django/Jinja transition In-Reply-To: <BANLkTikvghHWqXpnDQPKs-u8E4AH7XjEqw@mail.gmail.com> References: <BANLkTik5PURsjjufC+VdL2zgiCOKobA=Nw@mail.gmail.com> <BANLkTinptZ-SNywGichZfKJz_49fTgT8yw@mail.gmail.com> <BANLkTikvghHWqXpnDQPKs-u8E4AH7XjEqw@mail.gmail.com> Message-ID: <BANLkTikv1gjD6nZucW8hjqWWn+DSTuKnKg@mail.gmail.com> On 1 July 2011 16:30, anatoly techtonik <techtonik at gmail.com> wrote: > On Fri, Jul 1, 2011 at 3:01 AM, Richard Jones <richard at python.org> wrote: >> On 30 June 2011 04:57, anatoly techtonik <techtonik at gmail.com> wrote: >>> Index: pypi/templates/standard_template.pt >>> =================================================================== >>> - ? ? ?<title tal:content="string:Python Package Index : ${data/title}" /> >>> + ? ? ?<title tal:content="string:${data/title} : Python Package Index" /> >> >> Seems like a good idea. I've double-checked that the easy_install >> "API" doesn't depend on this HTML. Done. > > Have you seen the second patch? It should be more cool. I've reattached it. I don't believe it's necessary, thanks. The site is consistent with the code as above. Richard From richard at python.org Fri Jul 1 02:01:10 2011 From: richard at python.org (Richard Jones) Date: Fri, 1 Jul 2011 10:01:10 +1000 Subject: [Catalog-sig] Tab Title customization and Tal to Django/Jinja transition In-Reply-To: <BANLkTik5PURsjjufC+VdL2zgiCOKobA=Nw@mail.gmail.com> References: <BANLkTik5PURsjjufC+VdL2zgiCOKobA=Nw@mail.gmail.com> Message-ID: <BANLkTinptZ-SNywGichZfKJz_49fTgT8yw@mail.gmail.com> On 30 June 2011 04:57, anatoly techtonik <techtonik at gmail.com> wrote: > Index: pypi/templates/standard_template.pt > =================================================================== > - ? ? ?<title tal:content="string:Python Package Index : ${data/title}" /> > + ? ? ?<title tal:content="string:${data/title} : Python Package Index" /> Seems like a good idea. I've double-checked that the easy_install "API" doesn't depend on this HTML. Done. Can we stop the rather pointless arguing over templating engines now? Richard From merwok at netwok.org Sat Jul 2 14:18:08 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 02 Jul 2011 14:18:08 +0200 Subject: [Catalog-sig] Tab Title customization and Tal to Django/Jinja transition In-Reply-To: <BANLkTinptZ-SNywGichZfKJz_49fTgT8yw@mail.gmail.com> References: <BANLkTik5PURsjjufC+VdL2zgiCOKobA=Nw@mail.gmail.com> <BANLkTinptZ-SNywGichZfKJz_49fTgT8yw@mail.gmail.com> Message-ID: <4E0F0C80.3060007@netwok.org> Le 01/07/2011 02:01, Richard Jones a ?crit : > Seems like a good idea. I've double-checked that the easy_install > "API" doesn't depend on this HTML. Done. Thanks Richard! still-finding-that-the-colon-is-strange?ly yours, ?ric From faassen at startifact.com Mon Jul 4 17:41:17 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 04 Jul 2011 17:41:17 +0200 Subject: [Catalog-sig] disallowing the removal of packages? Message-ID: <iusmso$6nl$1@dough.gmane.org> Hi there, My development project relied on a certain version of a package that is hosted on pypi. Unfortunately the package mainatainer had decided that certain older versions of the package were not longer supported and had removed them from PyPI altogether. This broke the build procedure for my project: I had carefully pinned down that I wanted this version (as that's what we tested with), and it wasn't available anymore. So, the removal of old packages from PyPI means that other people who rely on these packages with their installation instructions ("install Foo 3.3") or build tools (installation instructions, automated), suddenly have to deal with breakage. I thought perhaps PyPI can help and handle this problem at the source. What are the possible use cases for removing a release from PyPI? a) mistakes: the upload was accidental - we weren't supposed to release this at all! I.e. I uploaded private code that I never meant to distribute in the first place. b) actively hostile: the upload actually contains deliberately malicious code and protecting people against this *should* break their builds c) legal issues: some party claims that this code is not PyPI's to distribute in the first place, and for legal issues it'd need to be taken down. d) broken: security or other bugs that makes us want to loudly say, this is so broken, don't use it anymore e) cleaning up old stale unsupported stuff. I'd argue d) and e) are not up to the package maintainer to decide but to the person who integrates this package into their system. The person who integrates the package is the one who will need to make the judgment call to continue to use old unsupported or broken stuff. Integrators should be allowed to make such decisions in their own time at their own convenience; the package developer shouldn't be able to force such decisions by removing an old release. (We can think about better warning mechanisms: perhaps there could be some metadata or rule to decide when an old release is "deprecated" and then integration and installation tools could warn about this, but all this would be to help the integrator a better decision on what to do.) Use cases a), b) and c) are left. I think b) and c) should be handled by the PyPI administrators: this takes a judgment call and the package maintainer might not be involved in this at all. Use case a) is the tricky one. I could upload something by mistake, and then immediately discover it and decide to throw it out again. But I'd be fine if I weren't allowed removing a release anymore after a period of a month. It could be that I only discover my mistake after a month, when people have already started relying on this version, but in that case as a PyPI user I'd want the administrators to issue a judgement call there too. So, my proposal would be to allow people to remove a recent release freely, but after a period of (I don't know? A day? A week? A month?) removing the old release is not allowed anymore. What do people think? Regards, Martijn From jim at zope.com Mon Jul 4 19:04:49 2011 From: jim at zope.com (Jim Fulton) Date: Mon, 4 Jul 2011 13:04:49 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iusmso$6nl$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> Message-ID: <CAPDm-Fj9UTTyB_pgcxpadocmhCRneKKtgh9yKFqB1B=xMBEo2g@mail.gmail.com> On Mon, Jul 4, 2011 at 11:41 AM, Martijn Faassen <faassen at startifact.com> wrote: > Hi there, > > My development project relied on a certain version of a package that is > hosted on pypi. Unfortunately the package mainatainer had decided that > certain older versions of the package were not longer supported and had > removed them from PyPI altogether. > > This broke the build procedure for my project: I had carefully pinned down > that I wanted this version (as that's what we tested with), and it wasn't > available anymore. > > So, the removal of old packages from PyPI means that other people who rely > on these packages with their installation instructions ("install Foo 3.3") > or build tools (installation instructions, automated), suddenly have to deal > with breakage. > > I thought perhaps PyPI can help and handle this problem at the source. > > What are the possible use cases for removing a release from PyPI? > > a) mistakes: the upload was accidental - we weren't supposed to release this > at all! I.e. I uploaded private code that I never meant to distribute in the > first place. > > b) actively hostile: the upload actually contains deliberately malicious > code and protecting people against this *should* break their builds > > c) legal issues: some party claims that this code is not PyPI's to > distribute in the first place, and for legal issues it'd need to be taken > down. > > d) broken: security or other bugs that makes us want to loudly say, this is > so broken, don't use it anymore > > e) cleaning up old stale unsupported stuff. > > I'd argue d) and e) are not up to the package maintainer to decide but to > the person who integrates this package into their system. The person who > integrates the package is the one who will need to make the judgment call to > continue to use old unsupported or broken stuff. Integrators should be > allowed to make such decisions in their own time at their own convenience; > the package developer shouldn't be able to force such decisions by removing > an old release. > > (We can think about better warning mechanisms: perhaps there could be some > metadata or rule to decide when an old release is "deprecated" and then > integration and installation tools could warn about this, but all this would > be to help the integrator a better decision on what to do.) > > Use cases a), b) and c) are left. I think b) and c) should be handled by the > PyPI administrators: this takes a judgment call and the package maintainer > might not be involved in this at all. > > Use case a) is the tricky one. I could upload something by mistake, and then > immediately discover it and decide to throw it out again. But I'd be fine if > I weren't allowed removing a release anymore after a period of a month. It > could be that I only discover my mistake after a month, when people have > already started relying on this version, but in that case as a PyPI user I'd > want the administrators to issue a judgement call there too. > > So, my proposal would be to allow people to remove a recent release freely, > but after a period of (I don't know? A day? A week? A month?) removing the > old release is not allowed anymore. > > What do people think? +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From pje at telecommunity.com Mon Jul 4 20:59:21 2011 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 04 Jul 2011 14:59:21 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iusmso$6nl$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> Message-ID: <20110704185935.71A623A404D@sparrow.telecommunity.com> At 05:41 PM 7/4/2011 +0200, Martijn Faassen wrote: >I'd argue d) and e) are not up to the package maintainer to decide >but to the person who integrates this package into their system. Sure... but that doesn't mean the package maintainer is obligated to support the integrator in that decision. >The person who integrates the package is the one who will need to >make the judgment call to continue to use old unsupported or broken >stuff. Integrators should be allowed to make such decisions in their >own time at their own convenience; the package developer shouldn't >be able to force such decisions by removing an old release. Then the package integrator should darn well keep their own copy instead of relying on it still being downloadable from a public server. Not keeping a file uploaded is not equal to forcing anybody else to do anything. Note that there is nothing in your proposal that keeps a package maintainer from simply never uploading packages to PyPI in the first place... and is likely to have the perverse effect of encouraging package authors who are concerned about this issue to make other hosting arrangements. (Certainly, if it looks like your proposal will be adopted, I would be strongly motivated to *immediately* remove any package from PyPI that I thought I might need to remove later, but would be unable to if the proposal were implemented!) In short, this proposal is asking PyPI to do a job that is properly done by either the web archive or your private backups. -1. From g.brandl at gmx.net Mon Jul 4 21:25:36 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 04 Jul 2011 21:25:36 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <20110704185935.71A623A404D@sparrow.telecommunity.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> Message-ID: <iut42g$l2b$1@dough.gmane.org> Am 04.07.2011 20:59, schrieb P.J. Eby: > At 05:41 PM 7/4/2011 +0200, Martijn Faassen wrote: >>I'd argue d) and e) are not up to the package maintainer to decide >>but to the person who integrates this package into their system. > > Sure... but that doesn't mean the package maintainer is obligated to > support the integrator in that decision. > > >>The person who integrates the package is the one who will need to >>make the judgment call to continue to use old unsupported or broken >>stuff. Integrators should be allowed to make such decisions in their >>own time at their own convenience; the package developer shouldn't >>be able to force such decisions by removing an old release. > > Then the package integrator should darn well keep their own copy > instead of relying on it still being downloadable from a public > server. Not keeping a file uploaded is not equal to forcing anybody > else to do anything. > > Note that there is nothing in your proposal that keeps a package > maintainer from simply never uploading packages to PyPI in the first > place... and is likely to have the perverse effect of encouraging > package authors who are concerned about this issue to make other > hosting arrangements. > > (Certainly, if it looks like your proposal will be adopted, I would > be strongly motivated to *immediately* remove any package from PyPI > that I thought I might need to remove later, but would be unable to > if the proposal were implemented!) > > In short, this proposal is asking PyPI to do a job that is properly > done by either the web archive or your private backups. -1. I have to agree. Any hosting service that doesn't allow me to remove uploaded content looks extremely fishy to me. I think this list has already discussed the issue of people using PyPI as a dependency for deployment in production, and recommended the use of private package mirrors. Georg From jacob at jacobian.org Mon Jul 4 21:34:45 2011 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Mon, 4 Jul 2011 14:34:45 -0500 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iut42g$l2b$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> Message-ID: <CAK8PqJE2W9bfDXROQE0XPmfW0s=u7karC-UdqCg7rizgPJfvdA@mail.gmail.com> On Mon, Jul 4, 2011 at 2:25 PM, Georg Brandl <g.brandl at gmx.net> wrote: > I have to agree. ?Any hosting service that doesn't allow me to remove > uploaded content looks extremely fishy to me. It pains me, but I have to agree as well. If I had a nickel for each time I've been bitten by a package disappearing off PyPI... well, I'd be able to buy a few beers, at least. I believe, as I've argued in the past, that PyPI's raison d'?tre is to be a cataloging resource *for package authors* ? if we want to prevent fragmentation, then we have to make PyPI the "path of least resistance" for packagers. This means PyPI can't be in the business of making these sorts of decisions on authors' behalf. Not that we shouldn't get upset when a package vanishes. I'd be all for some public chastisement in those cases... (80% joking). Jacob From fuzzyman at gmail.com Mon Jul 4 21:48:38 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Mon, 4 Jul 2011 20:48:38 +0100 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iut42g$l2b$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> Message-ID: <CAKCKLWzKac6YBeHSDuASgyi2sFjux+GLmgOyAt+PY2pthqx28A@mail.gmail.com> On 4 July 2011 20:25, Georg Brandl <g.brandl at gmx.net> wrote: > Am 04.07.2011 20:59, schrieb P.J. Eby: > > At 05:41 PM 7/4/2011 +0200, Martijn Faassen wrote: > >>I'd argue d) and e) are not up to the package maintainer to decide > >>but to the person who integrates this package into their system. > > > > Sure... but that doesn't mean the package maintainer is obligated to > > support the integrator in that decision. > > > > > >>The person who integrates the package is the one who will need to > >>make the judgment call to continue to use old unsupported or broken > >>stuff. Integrators should be allowed to make such decisions in their > >>own time at their own convenience; the package developer shouldn't > >>be able to force such decisions by removing an old release. > > > > Then the package integrator should darn well keep their own copy > > instead of relying on it still being downloadable from a public > > server. Not keeping a file uploaded is not equal to forcing anybody > > else to do anything. > > > > Note that there is nothing in your proposal that keeps a package > > maintainer from simply never uploading packages to PyPI in the first > > place... and is likely to have the perverse effect of encouraging > > package authors who are concerned about this issue to make other > > hosting arrangements. > > > > (Certainly, if it looks like your proposal will be adopted, I would > > be strongly motivated to *immediately* remove any package from PyPI > > that I thought I might need to remove later, but would be unable to > > if the proposal were implemented!) > > > > In short, this proposal is asking PyPI to do a job that is properly > > done by either the web archive or your private backups. -1. > > I have to agree. Any hosting service that doesn't allow me to remove > uploaded content looks extremely fishy to me. > > Sourceforge for a long time had a policy against removing any released source packages (and may still have that policy as far as I know). All the best, Michael > I think this list has already discussed the issue of people using PyPI > as a dependency for deployment in production, and recommended the use > of private package mirrors. > > Georg > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110704/7d6438f9/attachment-0001.html> From faassen at startifact.com Mon Jul 4 21:56:15 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 4 Jul 2011 21:56:15 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <20110704185935.71A623A404D@sparrow.telecommunity.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> Message-ID: <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> Hi there, If I hand a project that relies on 100 dependencies to someone, I prefer not to have to hand private copies of all those 100 dependencies to that someone as well just in case a package maintainer removes a version from PyPI. It just makes my life harder. I'd prefer to rely on a centrally maintained infrastructure in that case. I could, in theory, make my own backup arrangement where I basically replicate PyPI to keep all previous releases online forever, and then rely on that. If you place your packages on PyPI, your packages would end up on this backup arrangement of mine. But it seems odd not to do the right thing at the source. So came to discuss it there. So anyway, I'm discussing use cases. Let's get back to that. I think you have a very different view of what PyPI is for, or could be for, than I do. Is PyPI a service for Python developers to find reusable code? Is PyPI a hosting site for Python developers to publish their code online? Does PyPI support integrators? Or is it more like a hosting site where people can do whatever they want? How much is this like, say, Debian and how much is this like a developer's website? > (Certainly, if it looks like your proposal will be adopted, I would be strongly motivated to *immediately* remove any package from PyPI > that I thought I might need to remove later, but would be unable to if the proposal were implemented!) [why taking such an aggressive stance?] Clearly you have a concept of which packages or releases you might need to remove later: could you state your motivations behind removing packages or releases? Perhaps there's a use case I missed in the above, or perhaps, again, you have a different philosophy of what PyPI is for. Regards, Martijn From faassen at startifact.com Mon Jul 4 22:02:58 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 04 Jul 2011 22:02:58 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iut42g$l2b$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> Message-ID: <iut67b$1c0$1@dough.gmane.org> Hi there, On 07/04/2011 09:25 PM, Georg Brandl wrote: > I have to agree. Any hosting service that doesn't allow me to remove > uploaded content looks extremely fishy to me. That's the question: is PyPI like a web hosting service? Or is PyPI something else, or something else as well? Is wikipedia a hosting service, or something else? Should it be keeping old revisions of documents? Is Debian a hosting service, or something else? Should it be keeping old revisions of packages? > I think this list has already discussed the issue of people using PyPI > as a dependency for deployment in production, and recommended the use > of private package mirrors. Why should these package mirrors have to be private? Why can't we share one? Regards, Martijn From g.brandl at gmx.net Mon Jul 4 22:03:41 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 04 Jul 2011 22:03:41 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAKCKLWzKac6YBeHSDuASgyi2sFjux+GLmgOyAt+PY2pthqx28A@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> <CAKCKLWzKac6YBeHSDuASgyi2sFjux+GLmgOyAt+PY2pthqx28A@mail.gmail.com> Message-ID: <iut69o$17l$1@dough.gmane.org> Am 04.07.2011 21:48, schrieb Michael Foord: > > In short, this proposal is asking PyPI to do a job that is properly > > done by either the web archive or your private backups. -1. > > I have to agree. Any hosting service that doesn't allow me to remove > uploaded content looks extremely fishy to me. > > > > Sourceforge for a long time had a policy against removing any released source > packages (and may still have that policy as far as I know). I've just tried, I was able to remove all files from an old project (it is now hosted on PyPI only) just fine. Georg From faassen at startifact.com Mon Jul 4 22:06:45 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 04 Jul 2011 22:06:45 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAK8PqJE2W9bfDXROQE0XPmfW0s=u7karC-UdqCg7rizgPJfvdA@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> <CAK8PqJE2W9bfDXROQE0XPmfW0s=u7karC-UdqCg7rizgPJfvdA@mail.gmail.com> Message-ID: <iut6ee$3ce$1@dough.gmane.org> On 07/04/2011 09:34 PM, Jacob Kaplan-Moss wrote: > I believe, as I've argued in the past, that PyPI's raison > d'?tre is to be a cataloging resource *for package authors* ? if we > want to prevent fragmentation, then we have to make PyPI the "path of > least resistance" for packagers. This means PyPI can't be in the > business of making these sorts of decisions on authors' behalf. > > Not that we shouldn't get upset when a package vanishes. I'd be all > for some public chastisement in those cases... (80% joking). So can we then have *another* service running on python.org that keeps all older revisions of packages around, for the convience of *package users*? Because this is putting an extra burden on package users. And also on package authors, by the way: What if I release package foo, which needs package bar, version 2.1 or version 2.2. Now the author of package bar removes version 2.1 and 2.2. Now nobody can use package foo anymore. You can't argue that these people should've made a private mirror, as this applies to all *future* users of package foo either. This might be annoying to the author of foo... And the packager of foo, who *is* using a private mirror as you are all recommending, might not find out very quickly that this situation has emerged. :) Regards, Martijn From faassen at startifact.com Mon Jul 4 22:13:26 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 4 Jul 2011 22:13:26 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E121AAB.5010305@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> Message-ID: <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> Hi there, On Mon, Jul 4, 2011 at 9:55 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> What do people think? > > I agree with PJE, Jacob, and Georg: package owners need > to have absolute freedom to delete content at any time, > or even replace it with new content. Okay, if there's a consensus among PyPI maintainers that doing such things should be absolutely free to developers, what about a central backup server that does keep such old copies around that just clones PyPI? Evidently I'll have to find people who are interested in that. > I disagree about possible reactions, though: your first action > should be to ask the package author to bring the old version > back. Maybe they didn't know you were still using it. That was of course my first reaction, and that of several others in the same situation, a few weeks ago when this came up. One release was restored, but the release several of us were actually using wasn't. It was an easy upgrade, but I'd have preferred to prefer to deal with this situation at a time of my own choosing. So I figured I'd just prefer to use a system where such a situation was impossible, and it was clearly a problem others were having too, so can be solved centrally. So anyway, I'm dropping my proposal, as it's going nowhere. I'll submit another proposal in a few minutes. But I am genuinely curious about the use cases behind allowing package authors to have absolute freedom, by the way - there's something I am not understanding. Is this because it is thought that otherwise developers won't use PyPI at all? It's clear PJE is one such developer, but I'm trying to understand *why*. It can't be that it's considered good practice to change the contents of an older release, or to remove one, right? It seems positively dangerous to allow people to arbitrarily replace old packages with new content - installation instructions will be totally broken, and there are some security risks as well. And this freedom doesn't seem to offer much more control to developers, as once the packages are on PyPI, people can make copies elsewhere. So what is the motivation behind allowing this freedom? Purely the thought that developers otherwise won't want to use PyPI? But why do we want people to use PyPI in the first place if not to allow a convenient reuse of this code? I'm missing something here... Regards, Martijn From martin at v.loewis.de Mon Jul 4 21:55:23 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Jul 2011 21:55:23 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iusmso$6nl$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> Message-ID: <4E121AAB.5010305@v.loewis.de> > What do people think? I agree with PJE, Jacob, and Georg: package owners need to have absolute freedom to delete content at any time, or even replace it with new content. If you don't trust the author to keep old versions available, you have to make your own copy. Make sure you don't violate the usage agreement of the software when doing so. I disagree about possible reactions, though: your first action should be to ask the package author to bring the old version back. Maybe they didn't know you were still using it. Regards, Martin From faassen at startifact.com Mon Jul 4 22:31:31 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 04 Jul 2011 22:31:31 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI Message-ID: <iut7ss$bmc$1@dough.gmane.org> Hi there, Is there any interest in running an immutable mirror of PyPI on python.org as a service to package users? What it would do is mirror the PyPI index and packages, with one difference: releases and packages once mirrored will be mirrored indefinitely. It will not accept changes of existing releases, or removal of existing releases from the mirror. Instead, it would keep an archive of these forever. To deal with cases where people make an upload by mistake, there could be a "window of removal", however, where removal is accepted if a release is not older than a certain age. Is there perhaps already mirroring code that can be used to create such a service? The motivation is to share a service that many of us are using PyPI for already: a way to conveniently share packages without having to make local backups or distribute local copies to all people who use our project. To reliably share packages the current PyPI is not sufficient, as PyPI has a philosophy of being a hosting site for packagers and therefore should allow package maintainers to freely change or remove previous releases at any point in time. Such an immutable mirror would be useful to package developers as well: you can release package a that depends on package b. You can then know that package b can't just be removed or modified, so that people who download your package a from the mirror can be guaranteed to always have access to the same package b that you tested your code with yourself. There would need to be a mechanism for the mirror administrators to remove releases on rare occasions where this might be needed for reasons of security or legality. Regards, Martijn From lac at openend.se Mon Jul 4 22:39:12 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 04 Jul 2011 22:39:12 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: Message from Martijn Faassen <faassen@startifact.com> of "Mon, 04 Jul 2011 21:56:15 +0200." <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com><CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> Message-ID: <201107042039.p64KdCwP009787@theraft.openend.se> Idea: When somebody wants to remove a package, they get a message 'are you sure you want to do this? Other people may be relying on this package. Wouldn't it be kinder to them to give them a depreciation period?' And then, either kick in a depreciation period -- during which time if Martijn dl the package e gets told that it is deprecated, and is going away in XXX (reasonable time to be worked out later, for the purposes of this article, say 6 weeks). Then in 6 weeks the package automatically goes away. Or if the author says, "Hell yes, delete the thing now!" than that happens instead. Laura From faassen at startifact.com Mon Jul 4 22:49:19 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 4 Jul 2011 22:49:19 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <201107042039.p64KdCwP009787@theraft.openend.se> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <faassen@startifact.com> <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> <201107042039.p64KdCwP009787@theraft.openend.se> Message-ID: <CAGT4ZFgw0jvOmcgJ9PvhQhSb2kS5Z39=vShRBv-BJbrCqJFnMw@mail.gmail.com> Hi there, On Mon, Jul 4, 2011 at 10:39 PM, Laura Creighton <lac at openend.se> wrote: > When somebody wants to remove a package, they get a message > 'are you sure you want to do this? ?Other people may be relying > on this package. This would, I imagine, be an improvement. Anyway, it's not that this is happening a *lot*; it's in fact extremely rare. This is what makes it so tempting to use PyPI in the way I (and many others) are using it. Perhaps with a warning like this it would not happen at all anymore. But you can't rely on it. > Wouldn't it be kinder to them to give them > a depreciation period?' > ?And then, either kick in a depreciation > period -- during which time if Martijn dl the package e gets told > that it is deprecated, and is going away in XXX (reasonable time to > be worked out later, for the purposes of this article, say 6 weeks). > Then in 6 weeks the package automatically goes away. ?Or if the > author says, "Hell yes, delete the thing now!" than that happens > instead. A communication channel for package maintainers to tell package users "hey, this has a really serious security bug!" or "this is deprecated" would be useful. The package homepage on PyPI can be used for that, of course, though perhaps isn't perfect as people who are using your package indirectly might not ever see it. But even if we had a deprecation mechanism, it wouldn't exactly solve my use case, as it'd still mean that I would need to distribute my own backups of all packages I'm using to be able to reliably replicate an installation somewhere else. Sometimes this is not a burden, and often it's wise to do so anyway, but frequently (say with a bunch of geographically distributed developers) it is a lot more convenient to be able to rely on a central infrastructure. It's not like doing this for dependencies is unheard of either - people are relying on content distribution networks for Javascript dependencies, for instance. Anyway, not changing or removing old releases just seems like the Right Thing To Do in general in software development; I'd like the tools to support this. Regards, Martijn From mal at egenix.com Tue Jul 5 00:48:23 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 05 Jul 2011 00:48:23 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iut7ss$bmc$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> Message-ID: <4E124337.4020001@egenix.com> Martijn Faassen wrote: > Hi there, > > Is there any interest in running an immutable mirror of PyPI on > python.org as a service to package users? AFAIK, gocept is running such a mirror for the own purposes. You might want to partner up with them. I can put you in touch with Christian Theune/GoCept if you like. In general, I find the idea to use a potentially volatile service for running buildout or similar tools a hazardous approach to software configuration management, esp. in production environments. Why don't you just download the packages you have tested and ship them with your application, bypassing all the network and usability issues of a dynamic catalog server ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 05 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From pje at telecommunity.com Tue Jul 5 02:18:31 2011 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 04 Jul 2011 20:18:31 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <201107042039.p64KdCwP009787@theraft.openend.se> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> <201107042039.p64KdCwP009787@theraft.openend.se> Message-ID: <20110705001846.BF3FF3A404D@sparrow.telecommunity.com> At 10:39 PM 7/4/2011 +0200, Laura Creighton wrote: >Idea: > >When somebody wants to remove a package, they get a message >'are you sure you want to do this? Other people may be relying >on this package. Wouldn't it be kinder to them to give them >a depreciation period?' And then, either kick in a depreciation >period -- during which time if Martijn dl the package e gets told >that it is deprecated, and is going away in XXX (reasonable time to >be worked out later, for the purposes of this article, say 6 weeks). >Then in 6 weeks the package automatically goes away. Or if the >author says, "Hell yes, delete the thing now!" than that happens >instead. This would be nice if there were a way to communicate to automated installation tools about this deprecation, in such a way that it reached people who could do something about it... and somehow magically motivated them to act. In practice, such a message would be mostly ignored by users until it actually broke their builds, at which time they will actually pay attention to what it says on the PyPI page. (Or, more likely, they will complain to somebody upstream of them who will then look at the PyPI page.) To put it another way, if you are removing a package because you really don't want people using it unless they have an excellent and *considered* reason for doing so (e.g. a version with security issues), you're not going to get anybody to STOP using that version unless you break their mindless automated build. That's the only way you're going to get their (motivated) attention pointed at whatever needs changing. That doesn't mean deprecation isn't a good idea, though. Certainly, the build tools out there could be updated to flag warnings about deprecated packages, and people with projects big enough to have QA (i.e., the folks who most likely care about this issue anyway) probably WOULD pay attention to deprecation warnings. (I don't think I'd object to using a deprecation process as the normal way of removing a package, but on principle I believe an author should have the ability to remove their uploads for any reason or no reason.) From martin at v.loewis.de Tue Jul 5 09:38:54 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 09:38:54 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> Message-ID: <4E12BF8E.7060901@v.loewis.de> Am 04.07.2011 21:56, schrieb Martijn Faassen: > How much is this like, say, Debian > and how much is this like a developer's website? It's completely unlike Debian: - in Debian, you have package maintainers that perform the packaging, deal with bug reports (despite not being the original author), and decide which upstream release is put into what Debian release. Nobody takes that role in PyPI. - in Debian, you have the DFSG and other policy documents determining what licenses are allowed, and how software must behave. In PyPI, the only policy is that it must be a Python package. I dom't know what you mean by "developer's website", so I can't comment on whether PyPI is one. Regards, Martin From martin at v.loewis.de Tue Jul 5 09:51:53 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 09:51:53 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> Message-ID: <4E12C299.6000807@v.loewis.de> > But I am genuinely curious about the use cases behind allowing package > authors to have absolute freedom, by the way - there's something I am > not understanding. Is this because it is thought that otherwise > developers won't use PyPI at all? In general, PyPI prefers to not set policies when we can avoid it. There was a long discussion on what package names should be allowed (generally and also about specific package names), and the conclusion was always that package authors can put any content on PyPI as long as it's Python-related, and not spam or illegal. It's only consequential that there shouldn't be a policy on whether a package can be deleted. I *personally* think you are exaggerating your case. If you are using a package, avoid sticking to a specific version of the software, and instead write the package in a way that will work with many versions of the library. If the library is too unstable, don't use it at all. If the library has an incompatible change, report it as a bug; if the author can bring convincing arguments why there must be that change, cope with it. IOW, just don't use old versions when newer versions are available and maintained. > It can't be that it's considered good practice to change the contents > of an older release, or to remove one, right? Right. And it can't be considered good practice to rely on software that follows bad practice, right? Don't use software that deletes old releases while making unjustified incompatible changes in new releases, or otherwise doesn't provide clear backwards compatibility statements and migration strategies. > It seems positively > dangerous to allow people to arbitrarily replace old packages with new > content - installation instructions will be totally broken, and there > are some security risks as well. Again, you, as the package user, need to evaluate these risks. If you trust the author, you don't need to worry. If you don't trust the author, either don't use it, or take countermeasures (like inspecting all code before using it, and using hashsums in automated deployments). Regards, Martin From martin at v.loewis.de Tue Jul 5 09:55:15 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 09:55:15 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFgw0jvOmcgJ9PvhQhSb2kS5Z39=vShRBv-BJbrCqJFnMw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <faassen@startifact.com> <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> <201107042039.p64KdCwP009787@theraft.openend.se> <CAGT4ZFgw0jvOmcgJ9PvhQhSb2kS5Z39=vShRBv-BJbrCqJFnMw@mail.gmail.com> Message-ID: <4E12C363.90403@v.loewis.de> > A communication channel for package maintainers to tell package users > "hey, this has a really serious security bug!" or "this is deprecated" > would be useful. The package homepage on PyPI can be used for that, of > course, though perhaps isn't perfect as people who are using your > package indirectly might not ever see it. This may be a case where actually replacing an old release might be useful: you could put an actual DeprecationWarning into the code, or at least print a message in setup.py. This would increase the chance that anybody who has hard-coded the library version might see it. Regards, Martin From m.van.rees at zestsoftware.nl Tue Jul 5 13:16:39 2011 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Tue, 05 Jul 2011 13:16:39 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E12C299.6000807@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> Message-ID: <iuurqo$hd1$1@dough.gmane.org> Op 05-07-11 09:51, "Martin v. L?wis" schreef: > I *personally* think you are exaggerating your case. If you are using > a package, avoid sticking to a specific version of the software, and > instead write the package in a way that will work with many versions > of the library. If the library is too unstable, don't use it at all. > If the library has an incompatible change, report it as a bug; if > the author can bring convincing arguments why there must be that > change, cope with it. > > IOW, just don't use old versions when newer versions are available > and maintained. In my case, usually when I start a new (client) project for which I need package foo I use the latest release 1.0. The problems is that most of the time I do not know yet if a new version (whether that is going to be 1.0.1, 1.1 or 2.0) is going to cause backwards compatibility problems. And this may be the first time I am using package foo so I do not know yet if this author prefers to actively delete old versions. FWIW, I am using a pypi mirror created with collective.eggproxy as an index so that packages I use in my buildouts are available on that mirror (so only actually used packages are mirrored). This shields me at least partially from these problems. Cheers, -- Maurits van Rees Web App Programmer at Zest Software: http://zestsoftware.nl Personal website: http://maurits.vanrees.org/ From faassen at startifact.com Tue Jul 5 13:48:12 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 13:48:12 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E12C299.6000807@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> Message-ID: <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 9:51 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > I *personally* think you are exaggerating your case. If you are using > a package, avoid sticking to a specific version of the software, and > instead write the package in a way that will work with many versions > of the library. If the library is too unstable, don't use it at all. > If the library has an incompatible change, report it as a bug; if > the author can bring convincing arguments why there must be that > change, cope with it. I have about 5 years experience doing this. We started out doing it the way you suggested it in 2007. It didn't work. We got constant breakage. Users of your collection of packages would have to be lucky to download them on a day when it was all working perfectly. Everybody developing these packages were intelligent people working in good faith trying to preserve backwards compatibility, but things broke anyway. If you maintain the collection of packages you are willing to try upgrades to new versions of libraries and run the test suits to see whether it works, but if you then hand this collection of packages to other people that might start use it at unpredictable times, you want to hand them something that is guaranteed as much as possible to work. This applies to development teams when you just want to start working instead of having to deal with "oh this upgrade broke my build" all the time, and it also applies to open source projects which hand out a large collection of packages to users. [repeated] > I *personally* think you are exaggerating your case. ... > IOW, just don't use old versions when newer versions are available > and maintained. This leads me to conclude that I personally think you have not the same kind of experience with this issue as I do... I think you haven't been using packages on a scale I have been doing for years now. >> It can't be that it's considered good practice to change the contents >> of an older release, or to remove one, right? > > Right. And it can't be considered good practice to rely on software > that follows bad practice, right? I can't predict this in advance. I have been using this software for years, and suddenly it was removed. It's also rather facile of you to say that I can easily switch to something else when this happens - sometimes it's the only game in town. > Don't use software that deletes old releases while making unjustified > incompatible changes in new releases, or otherwise doesn't provide > clear backwards compatibility statements and migration strategies. Sorry, I can't see the future like you do. I also use packages that have imperfect maintainers that sometimes make incompatible releases by mistake; compatibility issues can be very subtle. Anyway, you're just telling me very loudly I don't have a problem, right? You're convinced I am doing it the wrong way. So should I just go away? Regards, Martijn From faassen at startifact.com Tue Jul 5 13:59:13 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 13:59:13 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E124337.4020001@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> Message-ID: <iuuu85$vmr$1@dough.gmane.org> Hi there, On 07/05/2011 12:48 AM, M.-A. Lemburg wrote: > Martijn Faassen wrote: >> Is there any interest in running an immutable mirror of PyPI on >> python.org as a service to package users? > > AFAIK, gocept is running such a mirror for the own purposes. > You might want to partner up with them. > > I can put you in touch with Christian Theune/GoCept if you like. I know Christian quite well, I can easily contact them. I hadn't realized they were running an immutable mirror. > In general, I find the idea to use a potentially volatile service > for running buildout or similar tools a hazardous approach to > software configuration management, esp. in production environments. > > Why don't you just download the packages you have tested and > ship them with your application, bypassing all the network > and usability issues of a dynamic catalog server ? Yeah, that's one way to resolve this. It's just a lot more work to do this during development than updating a version number in a configuration file when you need a new version. Concerning relying on networked resources for the installation of tested packages, Linux distributions have been doing this for years; I don't think that's a fundamentally flawed approach. PyPI just is at this point where it works 99.99% percent of the time, but it allows sudden surprises to pop up. Regards, Martijn From mal at egenix.com Tue Jul 5 14:12:41 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 05 Jul 2011 14:12:41 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iuuu85$vmr$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> Message-ID: <4E12FFB9.40603@egenix.com> Martijn Faassen wrote: > Hi there, > > On 07/05/2011 12:48 AM, M.-A. Lemburg wrote: >> Martijn Faassen wrote: >>> Is there any interest in running an immutable mirror of PyPI on >>> python.org as a service to package users? >> >> AFAIK, gocept is running such a mirror for the own purposes. >> You might want to partner up with them. >> >> I can put you in touch with Christian Theune/GoCept if you like. > > I know Christian quite well, I can easily contact them. I hadn't > realized they were running an immutable mirror. Ok. >> In general, I find the idea to use a potentially volatile service >> for running buildout or similar tools a hazardous approach to >> software configuration management, esp. in production environments. >> >> Why don't you just download the packages you have tested and >> ship them with your application, bypassing all the network >> and usability issues of a dynamic catalog server ? > > Yeah, that's one way to resolve this. > > It's just a lot more work to do this during development than updating a > version number in a configuration file when you need a new version. Hmm, the testing involved with upgrading to a new release is usually a lot more work than importing the release into a repository ;-) FWIW: We've been using the repository approach for many years now and it has never failed on us. Even doing a 1GB SVN checkout doesn't really take long if you're doing this on a server which is directly connected to the Internet. And after the checkout you can be sure that the production is running the exact same version as the one you've tested - a guarantee that tools such as buildout, which download the packages and then build them on the target machine, cannot provide. YMMV, though. > Concerning relying on networked resources for the installation of tested > packages, Linux distributions have been doing this for years; I don't > think that's a fundamentally flawed approach. Sure, but those are distributions of collections of packages that are known to work together (most of the time), not catalogs like PyPI, which don't provide any compatibility or availability guarantees. I was essentially suggesting to setup your own distribution or distribution server to avoid the issues with missing compatibility checks or loss of availability of a resource. > PyPI just is at this point where it works 99.99% percent of the time, > but it allows sudden surprises to pop up. I think that's just a coincident, not really a feature :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 05 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From faassen at startifact.com Tue Jul 5 15:14:26 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 15:14:26 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E12FFB9.40603@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> Message-ID: <CAGT4ZFgtgUamVh9Re-khmqB1_RDBKFAZ8NA4FtgJ_sXYV21gMw@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 2:12 PM, M.-A. Lemburg <mal at egenix.com> wrote: [snip] > Sure, but those are distributions of collections of packages > that are known to work together (most of the time), not catalogs > like PyPI, which don't provide any compatibility or availability > guarantees. > > I was essentially suggesting to setup your own distribution or > distribution server to avoid the issues with missing compatibility > checks or loss of availability of a resource. Yes, it's something to consider, and it's not too difficult to convert a project to a bunch of downloaded tarballs before deployment, but it's less friendly for sharing projects between developers; I'd have to maintain a separate distribution of lots of packages per project in my own version control system, once per project, right? Or I have to start maintaining my own distribution server. I don't want to maintain my own distribution server if I can share it. An immutable pypi would be a good distribution server, and mirroring is already practiced for PyPI and that would help with availability. It'd be a shared way to solve this problem as opposed to be everybody solving the problem for themselves. >> PyPI just is at this point where it works 99.99% percent of the time, >> but it allows sudden surprises to pop up. > > I think that's just a coincident, not really a feature :-) Yes, I realize that, that's why I initiated this discussion in the first place. It's only a few small steps away from a shared solution. Regards, Martijn From jim at zope.com Tue Jul 5 15:52:37 2011 From: jim at zope.com (Jim Fulton) Date: Tue, 5 Jul 2011 09:52:37 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E12FFB9.40603@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> Message-ID: <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> On Tue, Jul 5, 2011 at 8:12 AM, M.-A. Lemburg <mal at egenix.com> wrote: ... >> PyPI just is at this point where it works 99.99% percent of the time, >> but it allows sudden surprises to pop up. > > I think that's just a coincident, not really a feature :-) Actually, it's not. Within the Zope community, we early on realized the value of PyPI and setuptools as a collaboration tool and also realized that to fully realize it's potential, we'd need a policy of not removing old releases from PyPI. This policy was promulgated within the Zope community, has been followed, and has proven valuable. Of course, we still get burned from time to time when old versions of other packages we use disappear. I really don't intend to get embroiled in this debate, I've already offered a +1 to Martijn's proposal, but I will additionally offer a some opinions: - IMO, PyPI should be more for package consumers than for producers. I really have trouble fathoming the idea that a package index/catalog of primarily for producers. Perhaps I'm being dense. - A lot of PyPI's value is as a collaboration tool. It's presense, and more so with setuptools and its spawn, has made dynamic application assembly possible and reuse far more practical. - While I endorsed Martijn's idea, I think a voluntary system would be just as good and probably more palatable. I don't think we need a depracation system. There's rarely a good reason to remove an old package. Old releases are hidden from the human interface and automated tools will pick new releases unless told otherwise. I think if people understood that people were relying on old releases, they wouldn't remove them. IMO, it would be enough to warn someone about to delete that someone might be depending on the release and that deleting it could cause them hassle. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From faassen at startifact.com Tue Jul 5 16:18:54 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 16:18:54 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> Message-ID: <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.gmail.com> Hi there, Thanks for your input, Jim. > - IMO, PyPI should be more for package consumers than for producers. I > really have trouble fathoming the idea that a package index/catalog > of primarily for producers. Perhaps I'm being dense. Yes, I am being dense too, but then our perspective would be similar given our shared experiences. It seems clear from the responses by others that PyPI is seen as primarily for producers, not consumers of packages. That's why I asked whether PyPI is primarily a hosting site for developers (as opposed to something like Debian or Wikipedia which have notions about a collaborative effort of some kind, and care about preserving history). >From what I can interpret: PyPI is there so developers don't have to put the packages on their own web page, and to easily advertise that their packages and releases exist. Any use by consumers of PyPI for reuse of packages is secondary and really they should implement their own systems for reusing packages that does not rely on PyPI beyond an initial download. Is this a correct interpretation? Some more elaboration on the thinking behind this principle would be welcome. On Tue, Jul 5, 2011 at 3:52 PM, Jim Fulton <jim at zope.com> wrote: > ?IMO, it would be enough to warn someone about to delete that someone > ?might be depending on the release and that deleting it could cause > ?them hassle. Yes, this would indeed be helpful, as I expressed elsewhere. Regards, Martijn From lists at zopyx.com Tue Jul 5 16:25:40 2011 From: lists at zopyx.com (Andreas Jung) Date: Tue, 05 Jul 2011 16:25:40 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iusmso$6nl$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> Message-ID: <4E131EE4.2010700@zopyx.com> Martijn Faassen wrote: > Hi there, > > My development project relied on a certain version of a package that is > hosted on pypi. Unfortunately the package mainatainer had decided that > certain older versions of the package were not longer supported and had > removed them from PyPI altogether. > > This broke the build procedure for my project: I had carefully pinned > down that I wanted this version (as that's what we tested with), and it > wasn't available anymore. > > So, the removal of old packages from PyPI means that other people who > rely on these packages with their installation instructions ("install > Foo 3.3") or build tools (installation instructions, automated), > suddenly have to deal with breakage. > > I thought perhaps PyPI can help and handle this problem at the source. > > What are the possible use cases for removing a release from PyPI? > > a) mistakes: the upload was accidental - we weren't supposed to release > this at all! I.e. I uploaded private code that I never meant to > distribute in the first place. > > b) actively hostile: the upload actually contains deliberately malicious > code and protecting people against this *should* break their builds > > c) legal issues: some party claims that this code is not PyPI's to > distribute in the first place, and for legal issues it'd need to be > taken down. > > d) broken: security or other bugs that makes us want to loudly say, this > is so broken, don't use it anymore > > e) cleaning up old stale unsupported stuff. > > I'd argue d) and e) are not up to the package maintainer to decide but > to the person who integrates this package into their system. The person > who integrates the package is the one who will need to make the judgment > call to continue to use old unsupported or broken stuff. Integrators > should be allowed to make such decisions in their own time at their own > convenience; the package developer shouldn't be able to force such > decisions by removing an old release. > > (We can think about better warning mechanisms: perhaps there could be > some metadata or rule to decide when an old release is "deprecated" and > then integration and installation tools could warn about this, but all > this would be to help the integrator a better decision on what to do.) > > Use cases a), b) and c) are left. I think b) and c) should be handled by > the PyPI administrators: this takes a judgment call and the package > maintainer might not be involved in this at all. > > Use case a) is the tricky one. I could upload something by mistake, and > then immediately discover it and decide to throw it out again. But I'd > be fine if I weren't allowed removing a release anymore after a period > of a month. It could be that I only discover my mistake after a month, > when people have already started relying on this version, but in that > case as a PyPI user I'd want the administrators to issue a judgement > call there too. > > So, my proposal would be to allow people to remove a recent release > freely, but after a period of (I don't know? A day? A week? A month?) > removing the old release is not allowed anymore. First I have to second Martijn in all aspects (bascially because are coming from the same projects and facing similar problems). The main (meta) question is: is PyPI a the wild-west of the Python world where everyone can do with packages or do we want PyPI being a well-managed place for packages with a certain number of rules. Looking at the RSS feed each day makes me angry sometimes because PyPI feels like the dumpster of the Python world. The points Martijn raised are only one aspect of this question. -aj -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/a847a459/attachment.vcf> From lists at zopyx.com Tue Jul 5 16:35:51 2011 From: lists at zopyx.com (Andreas Jung) Date: Tue, 05 Jul 2011 16:35:51 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iut7ss$bmc$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> Message-ID: <4E132147.2040104@zopyx.com> Martijn Faassen wrote: > Hi there, > > Is there any interest in running an immutable mirror of PyPI on > python.org as a service to package users? Aren't the PyPI mirrors (based on top of the pep381 client) immutable? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/d6359920/attachment.vcf> From pje at telecommunity.com Tue Jul 5 16:42:56 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 05 Jul 2011 10:42:56 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.g mail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> Message-ID: <20110705144311.B7CD43A404D@sparrow.telecommunity.com> At 09:52 AM 7/5/2011 -0400, Jim Fulton wrote: >- IMO, PyPI should be more for package consumers than for producers. I > really have trouble fathoming the idea that a package index/catalog > of primarily for producers. Perhaps I'm being dense. The cardinal rule of successful groupware is that those who put information *into* the system must receive a proportional benefit for their efforts, or those who want to get information *out* of the system will not see anything coming out. From pje at telecommunity.com Tue Jul 5 16:51:03 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 05 Jul 2011 10:51:03 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.g mail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.gmail.com> Message-ID: <20110705145108.149DB3A404D@sparrow.telecommunity.com> At 04:18 PM 7/5/2011 +0200, Martijn Faassen wrote: >That's why I asked >whether PyPI is primarily a hosting site for developers (as opposed to >something like Debian or Wikipedia which have notions about a >collaborative effort of some kind, and care about preserving history). It's right there in the name: Python Package *Index*. In other words, it's an index of packages. Hosting the packages is optional. Heck, a package actually having any *code* to download at all is optional, since you can list a package you merely intend to develop. Or you might just have a revision control link, etc. So no, it's not a curated repository of code. If that were the case, it'd be better named the Python Code Repository or some such. The perspective that you have is influenced by your use case for PyPI; by nature, these other sorts of packages (i.e. ones without uploaded, released code) are ones you don't care about. But that doesn't mean they don't exist. In any case, you can't turn PyPI into PyCR by the simple expedient of disallowing deletions; you'd need actual curators and a fresh website that doesn't contain all the unreleased, unmaintained, un-existent, or un-open-source packages found on PyPI. From faassen at startifact.com Tue Jul 5 17:34:38 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 17:34:38 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110705145108.149DB3A404D@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.gmail.com> <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.g mail.com> <20110705145108.149DB3A404D@sparrow.telecommunity.com> Message-ID: <iuvas1$i38$1@dough.gmane.org> On 07/05/2011 04:51 PM, P.J. Eby wrote: > At 04:18 PM 7/5/2011 +0200, Martijn Faassen wrote: >> That's why I asked >> whether PyPI is primarily a hosting site for developers (as opposed to >> something like Debian or Wikipedia which have notions about a >> collaborative effort of some kind, and care about preserving history). > > It's right there in the name: Python Package *Index*. In other words, > it's an index of packages. Hosting the packages is optional. Heck, a > package actually having any *code* to download at all is optional, since > you can list a package you merely intend to develop. Or you might just > have a revision control link, etc. Something being an index doesn't necessarily imply that the index will behave in a certain way. For instance, if I have an index of a database table I rather expect that things mentioned in the index actually are kept in sync with things in the database table. It depends entirely on what kind of index it is trying to be. If PyPI's index followed some principles of immutability it could be used in different ways than if it isn't. > So no, it's not a curated repository of code. If that were the case, > it'd be better named the Python Code Repository or some such. Sure, if it turns out that is a more useful to people, it might be changed (given certain assumptions about what the word "index" applies) > The perspective that you have is influenced by your use case for PyPI; > by nature, these other sorts of packages (i.e. ones without uploaded, > released code) are ones you don't care about. But that doesn't mean they > don't exist. I'm not saying that my use case is the only use case for PyPI. I just submitted my use cases for discussion. This is why I was carefully analyzing use cases for package removal in the first place... I'm still missing a few, but people don't seem to want to share them. :) > In any case, you can't turn PyPI into PyCR by the simple expedient of > disallowing deletions; you'd need actual curators and a fresh website > that doesn't contain all the unreleased, unmaintained, un-existent, or > un-open-source packages found on PyPI. I think you misunderstand: I am the curator of my own list packages and versions. I just like using a shared central system by which those packages are shared (and many others I might want someday to add to my list). Using the existing PyPI repository for this works fairly well, but there are some issues, such as package changes and package removal. I thought that since others might have similar concerns, we could work together to offer some guarantees that would help external curators such as myself. It appears however that my use cases are not shared by most people responding in this thread. That doesn't mean that they're necessarily invalid use cases for PyPI, but I'm not holding out much hope after this reception. :) Regards, Martijn From faassen at startifact.com Tue Jul 5 17:38:47 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 17:38:47 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110705144311.B7CD43A404D@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.g mail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> Message-ID: <iuvb3p$js1$1@dough.gmane.org> On 07/05/2011 04:42 PM, P.J. Eby wrote: > At 09:52 AM 7/5/2011 -0400, Jim Fulton wrote: >> - IMO, PyPI should be more for package consumers than for producers. I >> really have trouble fathoming the idea that a package index/catalog >> of primarily for producers. Perhaps I'm being dense. > > The cardinal rule of successful groupware is that those who put > information *into* the system must receive a proportional benefit for > their efforts, or those who want to get information *out* of the system > will not see anything coming out. What's the proportional benefit for those who put stuff in the system? How much effort is involved in putting things into the system? How would immutability hinder the benefit? I don't understand. Regards, Martijn From faassen at startifact.com Tue Jul 5 17:42:51 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 17:42:51 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E132147.2040104@zopyx.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> Message-ID: <iuvbbe$lc2$1@dough.gmane.org> On 07/05/2011 04:35 PM, Andreas Jung wrote: > Martijn Faassen wrote: >> Is there any interest in running an immutable mirror of PyPI on >> python.org as a service to package users? > > > Aren't the PyPI mirrors (based on top of the pep381 client) > immutable? You tell me. Are they? How interesting. :) I think you're right! The package that disappeared from the main PyPI a few weeks ago is still available on b.pypi.python.org, at least. Wow, thanks! I had assumed the mirrors would remove things, but it seems they don't. I think that leaves the overwriting issue: if I were to release a version of foo 1.3, and then later upload a new version of foo 1.3, would the mirror be updated with the new version or not? Regards, Martijn From lists at zopyx.com Tue Jul 5 17:59:40 2011 From: lists at zopyx.com (Andreas Jung) Date: Tue, 05 Jul 2011 17:59:40 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iuvbbe$lc2$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> <iuvbbe$lc2$1@dough.gmane.org> Message-ID: <4E1334EC.5090101@zopyx.com> Martijn Faassen wrote: > On 07/05/2011 04:35 PM, Andreas Jung wrote: >> Martijn Faassen wrote: >>> Is there any interest in running an immutable mirror of PyPI on >>> python.org as a service to package users? >> >> >> Aren't the PyPI mirrors (based on top of the pep381 client) >> immutable? > > You tell me. Are they? How interesting. :) > > I think you're right! The package that disappeared from the main PyPI a > few weeks ago is still available on b.pypi.python.org, at least. > > Wow, thanks! :-P > > I had assumed the mirrors would remove things, but it seems they don't. > I think that leaves the overwriting issue: if I were to release a > version of foo 1.3, and then later upload a new version of foo 1.3, > would the mirror be updated with the new version or not? Not sure - MvL should know - never checked the pep381 sources in detail. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/2ab50fbf/attachment.vcf> From faassen at startifact.com Tue Jul 5 18:14:17 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 18:14:17 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E1334EC.5090101@zopyx.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> <iuvbbe$lc2$1@dough.gmane.org> <4E1334EC.5090101@zopyx.com> Message-ID: <iuvd6c$vq0$1@dough.gmane.org> Hi there, Hm, actually this is wrong. In fact, b.pypi.python.org appears to be seriously out of date, and c.pypi.python.org does seem to be synched. Regards, Martijn From faassen at startifact.com Tue Jul 5 18:20:24 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 18:20:24 +0200 Subject: [Catalog-sig] b mirror out of date? Message-ID: <iuvdhr$4ff$1@dough.gmane.org> Hi there, I just noticed that the 'b' mirror doesn't seem to be updated as much as the others. It does provide the 'last-modified' entry that other mirrors do, though it doesn't appear on the top-level index page like with the other mirrors (this is probably only an implementation detail). When I go there manually, I see: 20110303T12:42:55 Meaning it doesn't seem to have been updated in months. Is this intentional? Regards, Martijn From jim at zope.com Tue Jul 5 18:19:27 2011 From: jim at zope.com (Jim Fulton) Date: Tue, 5 Jul 2011 12:19:27 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110705145108.149DB3A404D@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.gmail.com> <20110705145108.149DB3A404D@sparrow.telecommunity.com> Message-ID: <CAPDm-Fj5iXh2SKE_4Ybi+1Y26LzXVKytgwJS4_NvNw0G5ETrxA@mail.gmail.com> On Tue, Jul 5, 2011 at 10:51 AM, P.J. Eby <pje at telecommunity.com> wrote: > At 04:18 PM 7/5/2011 +0200, Martijn Faassen wrote: >> >> That's why I asked >> whether PyPI is primarily a hosting site for developers (as opposed to >> something like Debian or Wikipedia which have notions about a >> collaborative effort of some kind, and care about preserving history). > > It's right there in the name: Python Package *Index*. ??In other words, it's > an index of packages. ??Hosting the packages is optional. ??Heck, a package > actually having any *code* to download at all is optional, since you can > list a package you merely intend to develop. ??Or you might just have a > revision control link, etc. Even if you buy that PyPI is *just* an index, when someone removes a release, they are removing the *index* data for it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From lists at zopyx.com Tue Jul 5 18:25:03 2011 From: lists at zopyx.com (Andreas Jung) Date: Tue, 05 Jul 2011 18:25:03 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iuvd6c$vq0$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> <iuvbbe$lc2$1@dough.gmane.org> <4E1334EC.5090101@zopyx.com> <iuvd6c$vq0$1@dough.gmane.org> Message-ID: <4E133ADF.3030501@zopyx.com> Martijn Faassen wrote: > Hi there, > > Hm, actually this is wrong. In fact, b.pypi.python.org appears to be > seriously out of date, and c.pypi.python.org does seem to be synched. which is running on my machine :) The root of the server shows you the last synchronization date. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/7e5b7850/attachment.vcf> From pydanny at gmail.com Tue Jul 5 19:33:19 2011 From: pydanny at gmail.com (Daniel Greenfeld) Date: Tue, 5 Jul 2011 10:33:19 -0700 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E131EE4.2010700@zopyx.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> Message-ID: <CAOoSJ_roGq4Dy1qSU445s_hKwt5g9gRuCFNFscpUOv2n6Wf2QQ@mail.gmail.com> Alright, my two cents: 1. I would love an eternal PyPI mirror. The only things that should go away are things that violate the law or have grave security risks. 2. You can't expect anyone on the planet to build applications that are robust enough to handle when a package or a particular release suddenly vanishes. For example, if I'm building a Bluebream app and zca.core goes away, then me and every other Zope developer is screwed. Same goes for PyQT, Django, and the rest of the Python universe. 3. IMHO, the index is for package maintainers AND consumers. Maintainers have a place that provides hosting and the ability to track usage, and consumers have a place that provides searching and a go-to place for install tools. If that wasn't the original intent during the inception of PyPI, then maybe the intent ought to be changed to match this new intention. Daniel Greenfeld On Tue, Jul 5, 2011 at 7:25 AM, Andreas Jung <lists at zopyx.com> wrote: > > > Martijn Faassen wrote: >> >> Hi there, >> >> My development project relied on a certain version of a package that is >> hosted on pypi. Unfortunately the package mainatainer had decided that >> certain older versions of the package were not longer supported and had >> removed them from PyPI altogether. >> >> This broke the build procedure for my project: I had carefully pinned >> down that I wanted this version (as that's what we tested with), and it >> wasn't available anymore. >> >> So, the removal of old packages from PyPI means that other people who >> rely on these packages with their installation instructions ("install >> Foo 3.3") or build tools (installation instructions, automated), >> suddenly have to deal with breakage. >> >> I thought perhaps PyPI can help and handle this problem at the source. >> >> What are the possible use cases for removing a release from PyPI? >> >> a) mistakes: the upload was accidental - we weren't supposed to release >> this at all! I.e. I uploaded private code that I never meant to >> distribute in the first place. >> >> b) actively hostile: the upload actually contains deliberately malicious >> code and protecting people against this *should* break their builds >> >> c) legal issues: some party claims that this code is not PyPI's to >> distribute in the first place, and for legal issues it'd need to be >> taken down. >> >> d) broken: security or other bugs that makes us want to loudly say, this >> is so broken, don't use it anymore >> >> e) cleaning up old stale unsupported stuff. >> >> I'd argue d) and e) are not up to the package maintainer to decide but >> to the person who integrates this package into their system. The person >> who integrates the package is the one who will need to make the judgment >> call to continue to use old unsupported or broken stuff. Integrators >> should be allowed to make such decisions in their own time at their own >> convenience; the package developer shouldn't be able to force such >> decisions by removing an old release. >> >> (We can think about better warning mechanisms: perhaps there could be >> some metadata or rule to decide when an old release is "deprecated" and >> then integration and installation tools could warn about this, but all >> this would be to help the integrator a better decision on what to do.) >> >> Use cases a), b) and c) are left. I think b) and c) should be handled by >> the PyPI administrators: this takes a judgment call and the package >> maintainer might not be involved in this at all. >> >> Use case a) is the tricky one. I could upload something by mistake, and >> then immediately discover it and decide to throw it out again. But I'd >> be fine if I weren't allowed removing a release anymore after a period >> of a month. It could be that I only discover my mistake after a month, >> when people have already started relying on this version, but in that >> case as a PyPI user I'd want the administrators to issue a judgement >> call there too. >> >> So, my proposal would be to allow people to remove a recent release >> freely, but after a period of (I don't know? A day? A week? A month?) >> removing the old release is not allowed anymore. > > First I have to second Martijn in all aspects (bascially because > are coming from the same projects and facing similar problems). > > The main (meta) question is: is PyPI a the wild-west of the Python world > where everyone can do with packages or do we want PyPI being a well-managed > place for packages with a certain number of rules. Looking at the RSS feed > each day makes me angry sometimes because PyPI feels like the dumpster of > the Python world. The points Martijn raised are only one aspect of this > question. > > -aj > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -- 'Knowledge is Power' Daniel Greenfeld http://pydanny.blogspot.com http://cartwheelweb.com From martin at v.loewis.de Tue Jul 5 19:46:10 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 19:46:10 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> Message-ID: <4E134DE2.2090704@v.loewis.de> > This leads me to conclude that I personally think you have not the > same kind of experience with this issue as I do... I think you haven't > been using packages on a scale I have been doing for years now. Perhaps. I typically use packages from Debian, and try to avoid downloading software from PyPI (or sourceforge or any other place where I need to investigate time to understand how the author does maintenance of hist software). I worry about upgrades every time Debian makes an upgrade, which is fairly infrequent. > Anyway, you're just telling me very loudly I don't have a problem, > right? Not at all. I can see the problem you are having. I'm just suggesting that you should take an entirely different approach to solving it, instead of the one you are heading to (i.e. make the download place provide packages "forever"). Regards, Martin From martin at v.loewis.de Tue Jul 5 19:51:07 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 19:51:07 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E131EE4.2010700@zopyx.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> Message-ID: <4E134F0B.4040007@v.loewis.de> > The main (meta) question is: is PyPI a the wild-west of the Python world > where everyone can do with packages Not sure what picture you have from the wild-west - in my (movie-taught) understanding, the picture would imply that anybody can do anything with *other* people's packages. That is certainly not the case for PyPI. Instead, PyPI is rather the small-town of the Python world: eeverybody can get a parcel of land, and then do on this land basically whatever they want. Business can open and close whenever they please. > or do we want PyPI being a well-managed place I think it's clear that PyPI won't ever be "managed", in any sense of the word, except to make sure that you can't stomp on somebody else's land (i.e. there is self-managed access control). Regards, Martin From martin at v.loewis.de Tue Jul 5 19:52:55 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 19:52:55 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAOoSJ_roGq4Dy1qSU445s_hKwt5g9gRuCFNFscpUOv2n6Wf2QQ@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <CAOoSJ_roGq4Dy1qSU445s_hKwt5g9gRuCFNFscpUOv2n6Wf2QQ@mail.gmail.com> Message-ID: <4E134F77.7040006@v.loewis.de> Am 05.07.2011 19:33, schrieb Daniel Greenfeld: > If that wasn't the original intent > during the inception of PyPI, then maybe the intent ought to be > changed to match this new intention. PyPI is certainly a place for user, i.e. where users can find Python software. The entire user interface is designed for that goal, as are the APIs. Regards, Martin From faassen at startifact.com Tue Jul 5 19:53:30 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 19:53:30 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E134DE2.2090704@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> Message-ID: <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 7:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Not at all. I can see the problem you are having. I'm just suggesting > that you should take an entirely different approach to solving it, > instead of the one you are heading to (i.e. make the download place > provide packages "forever"). Heading to? I've been doing this stuff since 2007. :) In a typical project I'm consuming 20 to a 200 packages, of which I'm maintaining myself perhaps 5 to 20%. The stuff I use typically isn't in Debian or has a version in Debian I don't want to use. It's working pretty well, but this was a small snag I had recently, and I thought, hey, let's see how to improve things. This is my secondary response a few weeks after the event: the first was of course to fix my project and to contact the author. I now think a special mirror might be the simplest way to solve this, since mirroring infrastructure already exist. Regards, Martijn From martin at v.loewis.de Tue Jul 5 20:00:40 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:00:40 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iuvb3p$js1$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.g mail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> Message-ID: <4E135148.7020204@v.loewis.de> > What's the proportional benefit for those who put stuff in the system? > How much effort is involved in putting things into the system? How would > immutability hinder the benefit? TO give a specific example: I'm getting tired of people asking for PyXML support, despite all pages where they could possibly get it from stating that it's deprecated and unmaintained. At times, I think the best solution would be to completely delete the software. Regards, Martin From faassen at startifact.com Tue Jul 5 19:57:00 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 19:57:00 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E134F0B.4040007@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> Message-ID: <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 7:51 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Instead, PyPI is rather the small-town of the Python world: eeverybody > can get a parcel of land, and then do on this land basically whatever > they want. Business can open and close whenever they please. Sounds like a hosting site for developers for me. The use case "everybody can do whatever they want" conflicts somewhat with the use case of being a system for others to access programmatically for package reuse. Anyway, to us city folk PyPI needs a few more guarantees; things that work in a small town don't always work in the city. >> or do we want PyPI being a well-managed place > > I think it's clear that PyPI won't ever be "managed", in any sense of > the word, except to make sure that you can't stomp on somebody else's > land (i.e. there is self-managed access control). What about making sure there's a historical record? No management built-into the system, but there is history. Regards, Martijn From martin at v.loewis.de Tue Jul 5 20:02:13 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:02:13 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E132147.2040104@zopyx.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> Message-ID: <4E1351A5.4060005@v.loewis.de> > Aren't the PyPI mirrors (based on top of the pep381 client) > immutable? No, they synchronize (including deletions). b is different since it runs on appengine, is broken, and I hadn't found the time to fix it yet. Regards, Martin From martin at v.loewis.de Tue Jul 5 20:03:09 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:03:09 +0200 Subject: [Catalog-sig] b mirror out of date? In-Reply-To: <iuvdhr$4ff$1@dough.gmane.org> References: <iuvdhr$4ff$1@dough.gmane.org> Message-ID: <4E1351DD.1050403@v.loewis.de> > Is this intentional? No, it's not. Nobody has found the time to fix it, yet. It runs on appengine, so it uses a different code base. Regards, Martin From martin at v.loewis.de Tue Jul 5 20:10:39 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:10:39 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> Message-ID: <4E13539F.20904@v.loewis.de> Am 05.07.2011 19:53, schrieb Martijn Faassen: > Hi there, > > On Tue, Jul 5, 2011 at 7:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Not at all. I can see the problem you are having. I'm just suggesting >> that you should take an entirely different approach to solving it, >> instead of the one you are heading to (i.e. make the download place >> provide packages "forever"). > > Heading to? I've been doing this stuff since 2007. :) I think you misunderstood: I was talking about the approach you are heading to (i.e. a download place that provides packages "forever"). ... unless I misunderstood, and you have been using such a mirror since 2007. > The stuff I use typically isn't in Debian or > has a version in Debian I don't want to use. I try to avoid software that is not packaged in Debian, and work around limitations that the versions in Debian may have. Regards, Martin From martin at v.loewis.de Tue Jul 5 20:11:28 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:11:28 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> Message-ID: <4E1353D0.70001@v.loewis.de> > What about making sure there's a historical record? No management > built-into the system, but there is history. There *is* a historical record, indeed. Regards, Martin From pydanny at gmail.com Tue Jul 5 20:12:12 2011 From: pydanny at gmail.com (Daniel Greenfeld) Date: Tue, 5 Jul 2011 11:12:12 -0700 Subject: [Catalog-sig] b mirror out of date? In-Reply-To: <4E1351DD.1050403@v.loewis.de> References: <iuvdhr$4ff$1@dough.gmane.org> <4E1351DD.1050403@v.loewis.de> Message-ID: <CAOoSJ_o33V71Jdr3SCGq2E4pPgAf3oUi2cOM6+REawCAwh4XOQ@mail.gmail.com> If it is so far out of date and no one has time to fix it, then maybe it should be taken off the mirror list? Danny On Tue, Jul 5, 2011 at 11:03 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Is this intentional? > > No, it's not. Nobody has found the time to fix it, yet. > It runs on appengine, so it uses a different code base. > > Regards, > Martin > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- 'Knowledge is Power' Daniel Greenfeld http://pydanny.blogspot.com http://cartwheelweb.com From faassen at startifact.com Tue Jul 5 20:24:21 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:24:21 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E135148.7020204@v.loewis.de> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> Message-ID: <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:00 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> What's the proportional benefit for those who put stuff in the system? >> How much effort is involved in putting things into the system? How would >> immutability hinder the benefit? > > TO give a specific example: I'm getting tired of people asking for PyXML > support, despite all pages where they could possibly get it from stating > that it's deprecated and unmaintained. At times, I think the best > solution would be to completely delete the software. An interesting example. I'm not convinced deleting PyXML would stop these people. :) We're talking about proportional benefit here, though. This sounds like a very rare case. It seems that "not wanting to support software" is somehow connected to "not providing the software for download" in people's minds - this is exactly the reason why the maintainer of the package that triggered this discussion for me removed these older versions. And as a result he probably got more support issues than if if he'd left them around. :) You can also see this as a historical record or version control issue. Just because version 0.7 is in a version control system somewhere doesn't mean that the developers are going to support this version. But in this case, I don't see developers argue there that this historical version should therefore be removed. With dependencies, my project in version control (or in released tarball form) has a dependency on an external system. I could instead check all dependencies into my version control system, or I could depend on an external system with more guarantees. That's the feedback I'm getting: either use a system with more guarantees such as a Debian release, or check everything into my own version control system. But this isn't always feasible. * debian might not have my dependencies available yet, or a different version, * I want to hack on the stable and development version of my project without having to install a virtual linux for each to make sure the debian dependencies are right * I want to automate the installation procedure, and therefore can't write an INSTALL.txt which says "apt-get this" * I want have a cross distribution compatible installation instructions Checking stuff into my own version control system has less issues and is indeed workable. There is a paradox there though: I want to release my package to PyPI in a reusable form and therefore can't include all Python dependencies inside it. So I'm not relying on PyPI to download packages myself but I'm asking consumers of my package to do so? Regards, Martijn From faassen at startifact.com Tue Jul 5 20:26:28 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:26:28 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E13539F.20904@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> Message-ID: <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:10 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Am 05.07.2011 19:53, schrieb Martijn Faassen: >> On Tue, Jul 5, 2011 at 7:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>> Not at all. I can see the problem you are having. I'm just suggesting >>> that you should take an entirely different approach to solving it, >>> instead of the one you are heading to (i.e. make the download place >>> provide packages "forever"). >> >> Heading to? I've been doing this stuff since 2007. :) > > I think you misunderstood: I was talking about the approach you are > heading to (i.e. a download place that provides packages "forever"). > ... unless I misunderstood, and you have been using such a mirror > since 2007. You're suggesting I completely rework the approach I have been using since 2007 to avoid breakage due to people removing releases. >> The stuff I use typically isn't in Debian or >> has a version in Debian I don't want to use. > > I try to avoid software that is not packaged in Debian, and > work around limitations that the versions in Debian may have. This works for a variety of use cases. But not for a whole bunch of other use cases. For instance: I need to deploy on a non-Debian platform. Now you again. :) Regards, Martijn From martin at v.loewis.de Tue Jul 5 20:26:40 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:26:40 +0200 Subject: [Catalog-sig] b mirror out of date? In-Reply-To: <CAOoSJ_o33V71Jdr3SCGq2E4pPgAf3oUi2cOM6+REawCAwh4XOQ@mail.gmail.com> References: <iuvdhr$4ff$1@dough.gmane.org> <4E1351DD.1050403@v.loewis.de> <CAOoSJ_o33V71Jdr3SCGq2E4pPgAf3oUi2cOM6+REawCAwh4XOQ@mail.gmail.com> Message-ID: <4E135760.4060207@v.loewis.de> Am 05.07.2011 20:12, schrieb Daniel Greenfeld: > If it is so far out of date and no one has time to fix it, then maybe > it should be taken off the mirror list? That shouldn't be necessary. Mirror clients should use the last-modified time stamp to find out that it is out of date. Regards, Martin From faassen at startifact.com Tue Jul 5 20:27:11 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:27:11 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E1353D0.70001@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> <4E1353D0.70001@v.loewis.de> Message-ID: <CAGT4ZFibgiv9DqWNTc4_MgfxP=9uAYSQL5-u9M5EDpYNRSRj1Q@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:11 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> What about making sure there's a historical record? No management >> built-into the system, but there is history. > > There *is* a historical record, indeed. Really? Where can I find this record? I.e. if you remove release 1.2 of foo, where can I find it again? Regards, Martijn From martin at v.loewis.de Tue Jul 5 20:42:30 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:42:30 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> Message-ID: <4E135B16.5070805@v.loewis.de> > It seems that "not wanting to support software" is somehow connected > to "not providing the software for download" in people's minds Indeed. As a package author, I find that highly plausible. If the software is no longer available anymore, people will stop using it, so they won't bother me with support requests. > this > is exactly the reason why the maintainer of the package that triggered > this discussion for me removed these older versions. And as a result > he probably got more support issues than if if he'd left them around. > :) That's only temporary. When the dust settles, people with either move to a newer version, or use something else altogether. > You can also see this as a historical record or version control issue. > Just because version 0.7 is in a version control system somewhere > doesn't mean that the developers are going to support this version. In free software, you have to support stuff when people contact you. So removal is the easiest way to reduce the amount of time you have to spend for support. > But in this case, I don't see developers argue there that this > historical version should therefore be removed. That's because the users that cause support effort don't check out the software from the repository - those that do check out old versions don't contact you for support. > With dependencies, my project in version control (or in released > tarball form) has a dependency on an external system. I could instead > check all dependencies into my version control system, or I could > depend on an external system with more guarantees. For example, you could depend on the source control system of the software you are depending on. > That's the feedback > I'm getting: either use a system with more guarantees such as a Debian > release, or check everything into my own version control system. But > this isn't always feasible. I can understand the problem. I'm just telling you that the solution you propose is an unacceptable interference with the freedom of the package authors (and this comes from somebody who thinks that a rating system is *not* an unacceptable interference with that freedom). > * debian might not have my dependencies available yet, or a different version, I won't give advice to you, just saying what I do: use the version in Debian. If it doesn't work, adjust your code until it does. If something isn't in Debian, I try to work without the package, possibly rewriting some of the functionality. If the amount of rewriting would be too large, I make an exception and install from PyPI. When doing so, I try to find a version whose dependencies can again be satisfied from PyPI. > * I want to hack on the stable and development version of my project > without having to install a virtual linux for each to make sure the > debian dependencies are right I try to depend only on released version of software, both for stable and development versions, and preferably on the same version of the dependency. So I can use a single Python installation. Regards, Martin From martin at v.loewis.de Tue Jul 5 20:44:16 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:44:16 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFibgiv9DqWNTc4_MgfxP=9uAYSQL5-u9M5EDpYNRSRj1Q@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> <4E1353D0.70001@v.loewis.de> <CAGT4ZFibgiv9DqWNTc4_MgfxP=9uAYSQL5-u9M5EDpYNRSRj1Q@mail.gmail.com> Message-ID: <4E135B80.8010904@v.loewis.de> Am 05.07.2011 20:27, schrieb Martijn Faassen: > Hi there, > > On Tue, Jul 5, 2011 at 8:11 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>> What about making sure there's a historical record? No management >>> built-into the system, but there is history. >> >> There *is* a historical record, indeed. > > Really? Where can I find this record? I.e. if you remove release 1.2 > of foo, where can I find it again? You can't find the release, but you can find a record of the date when the release was deleted. I can also find out who deleted it; this is not published, though. Regards, Martin From jim at zope.com Tue Jul 5 20:45:52 2011 From: jim at zope.com (Jim Fulton) Date: Tue, 5 Jul 2011 14:45:52 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> Message-ID: <CAPDm-FjJiY6Ok8mX2T5NhK9htsk=cDOWnwuM27WCYFZ1TFqERQ@mail.gmail.com> On Tue, Jul 5, 2011 at 1:53 PM, Martijn Faassen <faassen at startifact.com> wrote: ... > I now think a special mirror might be the simplest way to solve this, > since mirroring infrastructure already exist. Of course, it would have to be *special*, in that it would mirror additions and updates, but not removal. Also, ironically, this hurts developers because it makes it harder for them to remove things that *really* need to be removed. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From martin at v.loewis.de Tue Jul 5 20:46:43 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:46:43 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> Message-ID: <4E135C13.3010207@v.loewis.de> > On Tue, Jul 5, 2011 at 8:10 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Am 05.07.2011 19:53, schrieb Martijn Faassen: >>> On Tue, Jul 5, 2011 at 7:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>>> Not at all. I can see the problem you are having. I'm just suggesting >>>> that you should take an entirely different approach to solving it, >>>> instead of the one you are heading to (i.e. make the download place >>>> provide packages "forever"). >>> >>> Heading to? I've been doing this stuff since 2007. :) >> >> I think you misunderstood: I was talking about the approach you are >> heading to (i.e. a download place that provides packages "forever"). >> ... unless I misunderstood, and you have been using such a mirror >> since 2007. I take that back. I'm not proposing that you change anything about the way you work. Instead, I'm just reporting that there are completely alternative approaches possible at all which don't have the problem you are trying to solve. Regards, Martin From faassen at startifact.com Tue Jul 5 20:49:10 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:49:10 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E135B80.8010904@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> <4E1353D0.70001@v.loewis.de> <CAGT4ZFibgiv9DqWNTc4_MgfxP=9uAYSQL5-u9M5EDpYNRSRj1Q@mail.gmail.com> <4E135B80.8010904@v.loewis.de> Message-ID: <CAGT4ZFgB1ivtgTj7KNFneLZQr_3+P=JDCdjq4fmFSgBNkKv18Q@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:44 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Am 05.07.2011 20:27, schrieb Martijn Faassen: >> On Tue, Jul 5, 2011 at 8:11 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>>> What about making sure there's a historical record? No management >>>> built-into the system, but there is history. >>> >>> There *is* a historical record, indeed. >> >> Really? Where can I find this record? I.e. if you remove release 1.2 >> of foo, where can I find it again? > > You can't find the release, but you can find a record of the date when > the release was deleted. I can also find out who deleted it; this is > not published, though. Indeed a history of the index, not of the things being indexed, even if they're in PyPI itself. Regards, Martijn From faassen at startifact.com Tue Jul 5 20:51:35 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:51:35 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E135C13.3010207@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> <4E135C13.3010207@v.loewis.de> Message-ID: <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > I take that back. I'm not proposing that you change anything about > the way you work. Instead, I'm just reporting that there are completely > alternative approaches possible at all which don't have the problem you > are trying to solve. Sure. I was wrong to rely on PyPI to download and install the packages I need. Debian it is! :) Regards, Martijn From faassen at startifact.com Tue Jul 5 20:56:06 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:56:06 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E135B16.5070805@v.loewis.de> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> Message-ID: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:42 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: [snip] > I can understand the problem. I'm just telling you that the solution > you propose is an unacceptable interference with the freedom of the > package authors (and this comes from somebody who thinks that a rating > system is *not* an unacceptable interference with that freedom). Sure. Unacceptable interference with the freedom of package authors to remove stuff. Unacceptable because developers need the freedom to remove old versions of their software from the internet. :) So if someone comes up with a immutable mirror, do you think that's unacceptable interference with the freedom of package authors as well? Just checking. I'll just switch to debian for my package management needs. :) Regards, Martijn From ziade.tarek at gmail.com Tue Jul 5 23:50:30 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 5 Jul 2011 23:50:30 +0200 Subject: [Catalog-sig] b mirror out of date? In-Reply-To: <CAGSi+Q7Uqkap9jj=Q7mB+WLswjJnknWUfwSJhAR332CfJm8Ftw@mail.gmail.com> References: <iuvdhr$4ff$1@dough.gmane.org> <4E1351DD.1050403@v.loewis.de> <CAOoSJ_o33V71Jdr3SCGq2E4pPgAf3oUi2cOM6+REawCAwh4XOQ@mail.gmail.com> <4E135760.4060207@v.loewis.de> <CAGSi+Q7Uqkap9jj=Q7mB+WLswjJnknWUfwSJhAR332CfJm8Ftw@mail.gmail.com> Message-ID: <CAGSi+Q5OsiN5_+uHpSvvnYhTmY+QhT22FLpyY00gzM3c6siUXg@mail.gmail.com> Oops forgot to reply all Le 5 juil. 2011 23:48, "Tarek Ziad?" <ziade.tarek at gmail.com> a ?crit : > They should but some people are still picking a mirror by hand. I think a > good policy could be to remove mirrors which lag > 1 month > > Tarek > Le 5 juil. 2011 20:27, Martin v. L?wis <martin at v.loewis.de> a ?crit : >> Am 05.07.2011 20:12, schrieb Daniel Greenfeld: >>> If it is so far out of date and no one has time to fix it, then maybe >>> it should be taken off the mirror list? >> >> That shouldn't be necessary. Mirror clients should use the last-modified >> time stamp to find out that it is out of date. >> >> Regards, >> Martin >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/54cf67fb/attachment.html> From richard at python.org Tue Jul 5 23:55:54 2011 From: richard at python.org (Richard Jones) Date: Wed, 6 Jul 2011 07:55:54 +1000 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> <4E135C13.3010207@v.loewis.de> <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> Message-ID: <CAHrZfZB+DHR1LQmFmsgCyD0_b21hyQRdmDsG8YFiKTrcqwqMDw@mail.gmail.com> Could we rewind right back to the start of this conversation when it was suggested that you keep and ship a local mirror of packages you rely on? Such mirroring is trivial to set up. Richard From pje at telecommunity.com Wed Jul 6 03:19:27 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 05 Jul 2011 21:19:27 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.g mail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Message-ID: <20110706011938.0B6C13A4108@sparrow.telecommunity.com> At 08:56 PM 7/5/2011 +0200, Martijn Faassen wrote: >Sure. Unacceptable interference with the freedom of package authors to >remove stuff. Unacceptable because developers need >the freedom to remove old versions of their software from the internet. :) You still seem to be ignoring the part where insisting that people keep old things around for your convenience *also* means they get support requests from everyone *else* on the internet. In other words, a package author has to pay an (ongoing and perpetual) support cost for your temporary convenience. You are not making a principled moral stand here. Rather, you are asking that other people be made to carry an additional cost, for your benefit, on the basis that you don't think their cost is important, but you think your benefit is. From pje at telecommunity.com Wed Jul 6 03:23:11 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 05 Jul 2011 21:23:11 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAPDm-FjJiY6Ok8mX2T5NhK9htsk=cDOWnwuM27WCYFZ1TFqERQ@mail.g mail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <CAPDm-FjJiY6Ok8mX2T5NhK9htsk=cDOWnwuM27WCYFZ1TFqERQ@mail.gmail.com> Message-ID: <20110706012317.04C003A4108@sparrow.telecommunity.com> At 02:45 PM 7/5/2011 -0400, Jim Fulton wrote: >On Tue, Jul 5, 2011 at 1:53 PM, Martijn Faassen ><faassen at startifact.com> wrote: >... > > I now think a special mirror might be the simplest way to solve this, > > since mirroring infrastructure already exist. > >Of course, it would have to be *special*, in that it would mirror >additions and updates, but not removal. > >Also, ironically, this hurts developers because it makes it harder for >them to remove things that *really* need to be removed. I think it's well understood that you can't put the internet back in the bottle. But disallowing removals in the first place takes away the package author's right to decide what "really" means. In any case, the developer is no worse off than in the case where they have to convince the PyPI operators something "really" needs to be removed, and they are better off because at least the official distribution point for their package no longer carries the broken release. From lists at zopyx.com Wed Jul 6 06:20:03 2011 From: lists at zopyx.com (Andreas Jung) Date: Wed, 06 Jul 2011 06:20:03 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110706011938.0B6C13A4108@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <20110706011938.0B6C13A4108@sparrow.telecommunity.com> Message-ID: <4E13E273.8040803@zopyx.com> P.J. Eby wrote: > At 08:56 PM 7/5/2011 +0200, Martijn Faassen wrote: >> Sure. Unacceptable interference with the freedom of package authors to >> remove stuff. Unacceptable because developers need >> the freedom to remove old versions of their software from the >> internet. :) > > You still seem to be ignoring the part where insisting that people keep > old things around for your convenience *also* means they get support > requests from everyone *else* on the internet. In other words, a package > author has to pay an (ongoing and perpetual) support cost for your > temporary convenience. > > You are not making a principled moral stand here. Rather, you are asking > that other people be made to carry an additional cost, for your benefit, > on the basis that you don't think their cost is important, but you think > your benefit is. As a package author using PyPI for spreading my work for myself _and_ for other people implies that I take over a certain responsibility for - package quality and maturity - availability of my packages - reasonable metadata If a package author won't stick with those principles then a general hint to the author would be: stay away from PyPI and drop your package crap somewhere else. I don't care if other people see it different but PyPI must not be a place for autistic programmers that don't look left and right to the needs of package users and don't care about PyPI and general and that consider PyPI only their own private package sink. -aj -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110706/cd36eb73/attachment.vcf> From lists at zopyx.com Wed Jul 6 06:41:55 2011 From: lists at zopyx.com (Andreas Jung) Date: Wed, 06 Jul 2011 06:41:55 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E135B16.5070805@v.loewis.de> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> Message-ID: <4E13E793.5090904@zopyx.com> Martin v. L?wis wrote: > > I can understand the problem. I'm just telling you that the solution > you propose is an unacceptable interference with the freedom of the > package authors (and this comes from somebody who thinks that a rating > system is *not* an unacceptable interference with that freedom). Freedom of authors is one side of the medal. Playing nice in a community and sticking to some standard when using a community resource like PyPI is the other side. Freedom of authors does not stand over the needs of the community. Super-programmers not interested accepting certain rules still can do their own thing while leaving the community out of the game. -aj -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110706/e2f35c72/attachment.vcf> From martin at v.loewis.de Wed Jul 6 09:00:01 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Jul 2011 09:00:01 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Message-ID: <4E1407F1.5010105@v.loewis.de> > So if someone comes up with a immutable mirror, do you think that's > unacceptable interference with the freedom of package authors as well? I'm sure some package authors will object to such an installation. For the packages I maintain personally, doing so will be fine - I just don't want to be in charge of such a mirror, or have to defend it. Regards, Martin From mailing at franzoni.eu Wed Jul 6 09:59:16 2011 From: mailing at franzoni.eu (Alan Franzoni) Date: Wed, 6 Jul 2011 09:59:16 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Message-ID: <CAF3z5=mng+oocU0DTCDgA3r+r9tpRB_tmxHjn__e43hOsGnU-g@mail.gmail.com> On Tue, Jul 5, 2011 at 8:56 PM, Martijn Faassen <faassen at startifact.com> wrote: [cut] I had some thoughts myself on the topic; I think the idea of an immutable (or almost-immutable) pypi would be a rather good one. If you've had some kind of experience with Maven repositories in the Java world you know they're usually a good idea for dependency management; then you can setup your own internal proxy/mirror which you can rely on if some external server fades away. Mirroring a mutable pypi is not as easy as it sounds: what do I do if a release file is deleted and replaced with another file, same release number? How should my mirror work? Maven repos usually never allow for artifacts to be removed from their interface - you should remove them at filesystem level, but then you know you're doing something the system doesn't want you to. Rubygems approach is different but still takes care of version immutability a bit: you can yank a release, thus removing the artifact, but you're prevented from reuploading anything for the same release. The downside of mutability is: - today, I've got a working project which fetches dependencies through setuptools; - the project stays the same, one year passes. - The project no longer works because some deps on pypi changed or were removed. There're of course some good use cases for package removal, but they're "edgy": e.g. leaked credentials in source code, leaked unauthorized code, etc. OTOH there's the maintenance problem for package authors - I'd suggest something like a way to deprecate packages and releases, so that when anybody fetches such packages it can get a "deprecated" warning and should know the author is not maintaining such code anymore, but won't prevent legitimate and informed users to happily keep going with their existing software. Of course all of those things that I propose might just not fit into PyPI. If anybody is interested in creating a PyRepo or something like that, just let me know and I'd happily collaborate. -- http://www.franzoni.eu - public@[mysurname].eu Latest blog post: Unit testing with Twisted: testing protocols: http://t.co/HFpslG4 From mal at egenix.com Wed Jul 6 11:10:01 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Jul 2011 11:10:01 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Message-ID: <4E142669.7090307@egenix.com> Martijn Faassen wrote: > Hi there, > > On Tue, Jul 5, 2011 at 8:42 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > [snip] >> I can understand the problem. I'm just telling you that the solution >> you propose is an unacceptable interference with the freedom of the >> package authors (and this comes from somebody who thinks that a rating >> system is *not* an unacceptable interference with that freedom). > > Sure. Unacceptable interference with the freedom of package authors to > remove stuff. Unacceptable because developers need > the freedom to remove old versions of their software from the internet. :) > > So if someone comes up with a immutable mirror, do you think that's > unacceptable interference with the freedom of package authors as well? > Just checking. There are situations where you have to be able to remove a package file or even a complete package and by introducing public immutable servers you make it harder for package authors or PyPI maintainers to resolve such issues. Some possible reasons: * legal action (copyright, trademark, DMCA, license issues, etc) * removal of malicious packages (e.g. script kiddy stuff in setup.py) * reassigning package names (not sure whether that's possible with PyPI, but it certainly happens in the wild every now and then) * renaming packages (e.g. due to a poor initial package name choice) * seriously broken builds (e.g. that cause users to lose data) Private immutable index servers should not be a problem, since the package author or the PSF (as legal entity behind PyPI) has no direct way of fixing those. Please note that PyPI is getting bigger every day, so even if we are not seeing many such cases now, the problems can arise and if they do, they are going to cause a lot of trouble. Even as owner of a private immutable mirror, you would still want to know about such issues and I'm not sure whether you're really up to dealing with those cases as mirror maintainer. I'm repeating myself, but I still think you're better off creating a special mirror for your projects which then only has the packages and versions you want and need in your projects. Setting up such a mirror is really easy. All it takes is some web server, a directory with the package files and a script to create a static PyPI-style simple index file.... but you know all that, after all, Zope and Plone have been running such PyPI-style repositories for years :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 06 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mailing at franzoni.eu Wed Jul 6 16:20:52 2011 From: mailing at franzoni.eu (Alan Franzoni) Date: Wed, 6 Jul 2011 16:20:52 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E142669.7090307@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> Message-ID: <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> On Wed, Jul 6, 2011 at 11:10 AM, M.-A. Lemburg <mal at egenix.com> wrote: > Some possible reasons: > ?* renaming packages (e.g. due to a poor initial package name > ? choice) This is not a good reason, IMHO. You can go on with new versions and a new name, maybe you could want to deprecate the old package, but it's not a good reason to remove it. > ?* legal action (copyright, trademark, DMCA, license issues, etc) > > ?* removal of malicious packages (e.g. script kiddy stuff in > ? setup.py) > ?* seriously broken builds (e.g. that cause users to lose data) This is would make a good reason for package removal, but not for version reassignment. I.E. if I delete version 0.4.3 because it deleted my /usr instead of /usr/share/mylib/content, then I would be right at removing it, but there's no point in allowing any other package to take back 0.4.3. It's simply gone. > ?* reassigning package names (not sure whether that's possible with > ? PyPI, but it certainly happens in the wild every now and then) I'm not sure about what you mean here. BTW I think that the main issue here is *why* anybody should delete anything from pypi. If your conditions above apply, it's perfectly right for a dev to remove the packages. But I think most packages are removed without such conditions. Since I think such events are pretty rare, there could be a ticketing system dedicated to an "emergency removal" of artifacts, and that reupload of an artifact with the very same name-release should be inhibited. -- http://www.franzoni.eu - public@[mysurname].eu Latest blog post: Unit testing with Twisted: testing protocols: http://t.co/HFpslG4 From mal at egenix.com Wed Jul 6 17:54:04 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Jul 2011 17:54:04 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> Message-ID: <4E14851C.6070402@egenix.com> Please note that I'm talking about the idea of an immutable mirror, not PyPI itself - unfortunately, you've cut away that important context from my email. Alan Franzoni wrote: > On Wed, Jul 6, 2011 at 11:10 AM, M.-A. Lemburg <mal at egenix.com> wrote: >> Some possible reasons: > >> * renaming packages (e.g. due to a poor initial package name >> choice) > > This is not a good reason, IMHO. You can go on with new versions and a > new name, maybe you could want to deprecate the old package, but it's > not a good reason to remove it. I think undoing mistakes in package names is a very good reason to remove them. As package author you don't want such mistakes to stay on the net forever, if you can avoid it. >> * legal action (copyright, trademark, DMCA, license issues, etc) >> >> * removal of malicious packages (e.g. script kiddy stuff in >> setup.py) >> * seriously broken builds (e.g. that cause users to lose data) > > This is would make a good reason for package removal, but not for > version reassignment. I.E. if I delete version 0.4.3 because it > deleted my /usr instead of /usr/share/mylib/content, then I would be > right at removing it, but there's no point in allowing any other > package to take back 0.4.3. It's simply gone. I'm not sure I understand what you want to say. I wasn't talking about version reassignment in the above cases. >> * reassigning package names (not sure whether that's possible with >> PyPI, but it certainly happens in the wild every now and then) > > I'm not sure about what you mean here. Author A releases a package X, then drops the idea and removes the package, freeing up the name for others to use. Later on, author B uses the name X for something different and creates a new package X with a new set of releases. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 06 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From michael at voidspace.org.uk Fri Jul 8 21:22:40 2011 From: michael at voidspace.org.uk (Michael Foord) Date: Fri, 08 Jul 2011 20:22:40 +0100 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> Message-ID: <4E175900.6070907@voidspace.org.uk> Would it be possible to configure pypi to not recommend contacting webmaster at python.org <mailto:webmaster at python.org> on server error, but contact the pypi admins instead? All the best, Michael Foord -------- Original Message -------- Subject: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ Date: Fri, 8 Jul 2011 15:11:09 -0400 From: Jordan Goldstein <jordango at mit.edu> To: webmaster at python.org Enjoy. Please try again later. The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, webmaster at python.org <mailto:webmaster at python.org> and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. ------------------------------------------------------------------------ Apache/2.2.16 (Debian) Server at pypi.python.org <http://pypi.python.org> Port 80 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110708/8b8cf4d7/attachment.html> From sergey at maluke.com Sat Jul 9 17:14:02 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Sat, 9 Jul 2011 18:14:02 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi Message-ID: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> Hi, So there was some talk in paste-users list on releasing beta versions on PyPi or not, as easy_install and others pick them up by default. ( https://groups.google.com/d/msg/paste-users/1UAuadQjKng/qxVuF6lNU6QJ ) I've also had problems caused by that before (random segfaults due to beta release of lxml), so I can relate to the problem, but I still think betas should go to pypi. So, I've hacked together what seems like an obvious fix -- a pypi mirror/proxy that only includes stable versions. http://pypi-stable.maluke.com/ Anyway, just thought some people in this list might be interested in this. For more info see the thread linked above. -Sergey -- http://self.maluke.com/ From lists at zopyx.com Sun Jul 10 08:01:10 2011 From: lists at zopyx.com (Andreas Jung) Date: Sun, 10 Jul 2011 08:01:10 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> Message-ID: <4E194026.9040002@zopyx.com> Sorry but a pretty worthless effort since there is no enforced version schema and in addition the classifiers are often just unmaintained or wrong. -aj Sergey Schetinin wrote: > Hi, > > So there was some talk in paste-users list on releasing beta versions > on PyPi or not, as easy_install and others pick them up by default. ( > https://groups.google.com/d/msg/paste-users/1UAuadQjKng/qxVuF6lNU6QJ ) > I've also had problems caused by that before (random segfaults due to > beta release of lxml), so I can relate to the problem, but I still > think betas should go to pypi. > > So, I've hacked together what seems like an obvious fix -- a pypi > mirror/proxy that only includes stable versions. > http://pypi-stable.maluke.com/ > > Anyway, just thought some people in this list might be interested in this. > For more info see the thread linked above. > > -Sergey > -- ZOPYX Limited | zopyx group Charlottenstr. 37/1 | The full-service network for Zope & Plone D-72070 T?bingen | Produce & Publish www.zopyx.com | www.produce-and-publish.com ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110710/fb0bfae9/attachment.vcf> From ben+python at benfinney.id.au Sun Jul 10 09:10:08 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 10 Jul 2011 17:10:08 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> Message-ID: <87mxgmfw6n.fsf@benfinney.id.au> Andreas Jung <lists at zopyx.com> writes: > Sorry but a pretty worthless effort since there is no enforced version > schema and in addition the classifiers are often just unmaintained or > wrong. Conversely, the presence of a tool that uses the classifier information in this way could encourage people to put useful information in there. +1. I'm in favour, all else being equal, of having a distinct repository of stable-release-only packages. -- \ ?The Vatican is not a state.? a state must have people. There | `\ are no Vaticanians.? No-one gets born in the Vatican except by | _o__) an unfortunate accident.? ?Geoffrey Robertson, 2010-09-18 | Ben Finney From ziade.tarek at gmail.com Sun Jul 10 10:11:20 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 10 Jul 2011 10:11:20 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E194026.9040002@zopyx.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> Message-ID: <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> This is not true anymore for metadata 1.2 which follow pep 386. Pypi already implements it. Pushing a non-386 version gets your action rejected. Plus distutils 2 has a prefer final flag Cheers Tarek Le 10 juil. 2011 08:01, "Andreas Jung" <lists at zopyx.com> a ?crit : > Sorry but a pretty worthless effort since there is no enforced version > schema and in addition the classifiers are often just unmaintained or wrong. > > -aj > > Sergey Schetinin wrote: >> Hi, >> >> So there was some talk in paste-users list on releasing beta versions >> on PyPi or not, as easy_install and others pick them up by default. ( >> https://groups.google.com/d/msg/paste-users/1UAuadQjKng/qxVuF6lNU6QJ ) >> I've also had problems caused by that before (random segfaults due to >> beta release of lxml), so I can relate to the problem, but I still >> think betas should go to pypi. >> >> So, I've hacked together what seems like an obvious fix -- a pypi >> mirror/proxy that only includes stable versions. >> http://pypi-stable.maluke.com/ >> >> Anyway, just thought some people in this list might be interested in this. >> For more info see the thread linked above. >> >> -Sergey >> > > -- > ZOPYX Limited | zopyx group > Charlottenstr. 37/1 | The full-service network for Zope & Plone > D-72070 T?bingen | Produce & Publish > www.zopyx.com | www.produce-and-publish.com > ------------------------------------------------------------------------ > E-Publishing, Python, Zope & Plone development, Consulting > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110710/2cd26b80/attachment.html> From lists at zopyx.com Sun Jul 10 11:39:28 2011 From: lists at zopyx.com (Andreas Jung) Date: Sun, 10 Jul 2011 11:39:28 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> Message-ID: <4E197350.6040204@zopyx.com> Tarek Ziad? wrote: > This is not true anymore for metadata 1.2 which follow pep 386. Which is likely a very small number of packages. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110710/f23ad579/attachment.vcf> From sergey at maluke.com Sun Jul 10 16:24:12 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Sun, 10 Jul 2011 17:24:12 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E194026.9040002@zopyx.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> Message-ID: <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.gmail.com> On 10 July 2011 09:01, Andreas Jung <lists at zopyx.com> wrote: > Sorry but a pretty worthless effort since there is no enforced version > schema and in addition the classifiers are often just unmaintained or wrong. Sure. All it does is fix the problem right now, without telling anyone to change how they do releases etc. Worthless. If you know of any releases that are final but are filtered out in that mirror, that would be more constructive. -- http://self.maluke.com/ From pje at telecommunity.com Sun Jul 10 17:51:05 2011 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 10 Jul 2011 11:51:05 -0400 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.g mail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.gmail.com> Message-ID: <20110710155132.3C9243A409B@sparrow.telecommunity.com> At 05:24 PM 7/10/2011 +0300, Sergey Schetinin wrote: >If you know of any releases that are final but are filtered out in >that mirror, that would be more constructive. ISTM that you are currently filtering out packages whose *only* available versions are alpha, beta, candidate, etc. So if a package is still experimental or in development, it is not available. You might want to consider using a "most stable available release type" instead. That is, if a project only has alphas and betas, provide the betas. From sergey at maluke.com Sun Jul 10 17:53:38 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Sun, 10 Jul 2011 18:53:38 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <20110710155132.3C9243A409B@sparrow.telecommunity.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.gmail.com> <20110710155132.3C9243A409B@sparrow.telecommunity.com> Message-ID: <CAP2iL+C4Rw3ymYtqG6f7P0-WReBSGDB+zBHwVU5hUbmmQa5PBg@mail.gmail.com> On 10 July 2011 18:51, P.J. Eby <pje at telecommunity.com> wrote: > At 05:24 PM 7/10/2011 +0300, Sergey Schetinin wrote: >> >> If you know of any releases that are final but are filtered out in >> that mirror, that would be more constructive. > > ISTM that you are currently filtering out packages whose *only* available > versions are alpha, beta, candidate, etc. ?So if a package is still > experimental or in development, it is not available. > > You might want to consider using a "most stable available release type" > instead. ?That is, if a project only has alphas and betas, provide the > betas. Good point, thanks. -- http://self.maluke.com/ From sergey at maluke.com Sun Jul 10 18:31:43 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Sun, 10 Jul 2011 19:31:43 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+C4Rw3ymYtqG6f7P0-WReBSGDB+zBHwVU5hUbmmQa5PBg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.gmail.com> <20110710155132.3C9243A409B@sparrow.telecommunity.com> <CAP2iL+C4Rw3ymYtqG6f7P0-WReBSGDB+zBHwVU5hUbmmQa5PBg@mail.gmail.com> Message-ID: <CAP2iL+CU+=B1cge3Ofrc1Rtn8M5u2La7RcVZOcqKE3vzUDOcAQ@mail.gmail.com> So I did what Phillip suggested. Also I now don't filter out the less-stable releases completely -- I remove the links, but still list them, so it's easier to see what the proxy does. For example: http://pypi-stable.maluke.com/simple/setuptools/ http://pypi-stable.maluke.com/simple/lxml http://pypi-stable.maluke.com/simple/WebOb -- http://self.maluke.com/ From merwok at netwok.org Mon Jul 11 14:41:18 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 11 Jul 2011 14:41:18 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> Message-ID: <4E1AEF6E.9020700@netwok.org> Le 10/07/2011 10:11, Tarek Ziad? a ?crit : > This is not true anymore for metadata 1.2 which follow pep 386. Pypi already > implements it. Pushing a non-386 version gets your action rejected. I was under the impression that PEP 386 only defined the syntax of version numbers and a comparison algo, but no semantics. IOW there is no way for a tool to know that 2.6.33 is devel and 2.6.34 stable, or that 1.0.4 does not break compatibility with 1.0.2, or anything else of the sort. > Plus distutils 2 has a prefer final flag Is final the same thing as stable? Regards From tseaver at palladion.com Mon Jul 11 18:45:16 2011 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 11 Jul 2011 12:45:16 -0400 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1AEF6E.9020700@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> Message-ID: <ivf9aq$bgv$1@dough.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/11/2011 08:41 AM, ?ric Araujo wrote: > Le 10/07/2011 10:11, Tarek Ziad? a ?crit : >> This is not true anymore for metadata 1.2 which follow pep 386. >> Pypi already implements it. Pushing a non-386 version gets your >> action rejected. > > I was under the impression that PEP 386 only defined the syntax of > version numbers and a comparison algo, but no semantics. IOW there > is no way for a tool to know that 2.6.33 is devel and 2.6.34 stable, > or that 1.0.4 does not break compatibility with 1.0.2, or anything > else of the sort. The PEP386 semantics are implied by the sorting order, where "final" (non-suffixed) releases sort higher than any suffixed releases except "post" releases. Any suffix from the 'a|b|c|rc' group implies a "non-final" / "development" version of some kind (alpha, beta, release candidate), while suffixes in the 'post' group imply some kind of patch-level (presumed stable unless re-suffixed with 'dev'). Note that zc.buildout already has a 'prefer-final' option which uses these semantics: http://pypi.python.org/pypi/zc.buildout/1.5.2#preferring-final-releases Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4bKJwACgkQ+gerLs4ltQ4M4QCeK7p3jzqzuWl3VzZJJ8gyw7+L IM8AoLgr/1+l3tmbpBVBwF9nJKYt+QkD =K3oO -----END PGP SIGNATURE----- From ziade.tarek at gmail.com Mon Jul 11 19:42:16 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 19:42:16 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E197350.6040204@zopyx.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> Message-ID: <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> On Sun, Jul 10, 2011 at 11:39 AM, Andreas Jung <lists at zopyx.com> wrote: > > > Tarek Ziad? wrote: >> >> This is not true anymore for metadata 1.2 which follow pep 386. > > Which is likely a very small number of packages. Oh yeah, maybe none except mine right now :) but the point is: I think there's some value right now in adding PEP 386 support in existing tools + a prefer-final option in every installers, than trying to set up a new 'stable only' pypi It turns out that around 80% of PyPI packages are PEP 386 compliant already.. > > Andreas > -- Tarek Ziad? | http://ziade.org From sergey at maluke.com Mon Jul 11 20:33:45 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Mon, 11 Jul 2011 21:33:45 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> Message-ID: <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> On 11 July 2011 20:42, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > but the point is: I think there's some value right now in adding PEP > 386 support in existing tools + a prefer-final option in every > installers, than trying to set up a new 'stable only' pypi Well, if the data is there, it's super-easy to filter on the server-side. And why repeat that work in every installer if it can be accomplished by supplying a different pypi index url? -- http://self.maluke.com/ From ziade.tarek at gmail.com Mon Jul 11 20:43:57 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 20:43:57 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> Message-ID: <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> On Mon, Jul 11, 2011 at 8:33 PM, Sergey Schetinin <sergey at maluke.com> wrote: > On 11 July 2011 20:42, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >> but the point is: I think there's some value right now in adding PEP >> 386 support in existing tools + a prefer-final option in every >> installers, than trying to set up a new 'stable only' pypi > > Well, if the data is there, it's super-easy to filter on the > server-side. Is that a proxy project for you or do you want the community to create a new pypi server that filters out non-stable releases ? If it's the latter, what version scheme will you use to decide whether a version is alpha beta etc. ? It would make sense to use PEP 386. If so, the filter that you are creating can be hooked on client side as easily as server side, the only difference is that you don't have to change the existing infrastructure and deal with mirrors. (unless the proxy is to be run on client side ?) > And why repeat that work in every installer if it can be > accomplished by supplying a different pypi index url? Because ideally we would want all installers to behave the same way wrt versions and PyPI repositories. If I run a mirror of PyPI and I don't use your proxy, I don't have the "prefer final" version. If this is a client-side option it may work on any PyPI repo, mirror, or even on any directory containing packages (since pip and setuptools allow scanning local dirs) > > > -- > http://self.maluke.com/ > -- Tarek Ziad? | http://ziade.org From sergey at maluke.com Mon Jul 11 21:03:57 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Mon, 11 Jul 2011 22:03:57 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> Message-ID: <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> On 11 July 2011 21:43, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > On Mon, Jul 11, 2011 at 8:33 PM, Sergey Schetinin <sergey at maluke.com> wrote: >> On 11 July 2011 20:42, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >>> but the point is: I think there's some value right now in adding PEP >>> 386 support in existing tools + a prefer-final option in every >>> installers, than trying to set up a new 'stable only' pypi >> >> Well, if the data is there, it's super-easy to filter on the >> server-side. > > Is that a proxy project for you or do you want the community to create > a new pypi server that filters out non-stable releases ? > > If it's the latter, what version scheme will you use to decide whether > a version is alpha beta etc. ? ?It would make sense to use PEP 386. If > so, the filter that you are creating can be hooked on client side as > easily as server side, the only difference is that you don't have to > change the existing infrastructure and deal with mirrors. > > (unless the proxy is to be run on client side ?) That webapp (it's not *really* a proxy) took me about two hours total, and it has to query pypi, guess what is stable and not, cache the results. If there's no guessing and all data is local, how hard can it be? I'm not asking anyone to do it, but it does look like it would be a simpler, more immediate solution. My point is, the server can use whatever the most current metadata format is out there to make the decision, maybe falling back to guessing if the package does not include it, and it would work immediately, with every install tool that supports custom package indices. However I see that the mirrors issue makes this impractical. -- http://self.maluke.com/ From ziade.tarek at gmail.com Mon Jul 11 21:20:36 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 21:20:36 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> Message-ID: <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: ... > > That webapp (it's not *really* a proxy) took me about two hours total, > and it has to query pypi, guess what is stable and not, cache the > results. how do you guess it ? that's an interesting part > If there's no guessing and all data is local, how hard can it > be? I'm not asking anyone to do it, but it does look like it would be > a simpler, more immediate solution. istm that it can be complex to have it working over the mirrors, unless it's a proxy on client-side. But then we're closer to a client-side change I guess. > > My point is, the server can use whatever the most current metadata > format is out there to make the decision, maybe falling back to > guessing if the package does not in nclude it, and it would work > immediately, with every install tool that supports custom package > indices. So right now the 'version' field is used only -- and present in all metadata versions at PyPI. There are now three versions schemes out there to my knowledge, with small differences (see PEP 386 for the details) pep 386 (#3) is still irrelevant but will start to gain traction once 3.3 is out. Maybe there's a way to sort stable vs none stable versions across all schemes and versions.. > However I see that the mirrors issue makes this impractical. Yes that's the main concern > > -- > http://self.maluke.com/ > -- Tarek Ziad? | http://ziade.org From sergey at maluke.com Mon Jul 11 21:32:36 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Mon, 11 Jul 2011 22:32:36 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> Message-ID: <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> On 11 July 2011 22:20, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: > ... >> >> That webapp (it's not *really* a proxy) took me about two hours total, >> and it has to query pypi, guess what is stable and not, cache the >> results. > > how do you guess it ? that's an interesting part Well, I mentioned it in linked post. Basically at first I was doing just this: _rx_final = re.compile(r'[\d\.]+\Z') def is_final(ver): return bool(_rx_final.match(ver)) But to implement the most-stable filter, I changed it to this: http://pastebin.com/JVVAJ9hE >> If there's no guessing and all data is local, how hard can it >> be? I'm not asking anyone to do it, but it does look like it would be >> a simpler, more immediate solution. > > istm that it can be complex to have it working over the mirrors, > unless it's a proxy on client-side. > But then we're closer to a client-side change I guess. Given the existing mirrors infrastructure it is indeed hard. If the mirrors were demoted to just serving downloads and not the index, they could still be used with such an index as-is. As I said, I have to agree it would be impractical to add a stable-only pypi index and try to mirror it as well. -- http://self.maluke.com/ From ziade.tarek at gmail.com Mon Jul 11 21:47:29 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 21:47:29 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> Message-ID: <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> On Mon, Jul 11, 2011 at 9:32 PM, Sergey Schetinin <sergey at maluke.com> wrote: > On 11 July 2011 22:20, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >> On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: >> ... >>> >>> That webapp (it's not *really* a proxy) took me about two hours total, >>> and it has to query pypi, guess what is stable and not, cache the >>> results. >> >> how do you guess it ? that's an interesting part > > Well, I mentioned it in linked post. Basically at first I was doing just this: > > _rx_final = re.compile(r'[\d\.]+\Z') > > def is_final(ver): > ? ?return bool(_rx_final.match(ver)) > > But to implement the most-stable filter, I changed it to this: > http://pastebin.com/JVVAJ9hE there must be a bug because all the versions I am trying are marked stable (even 'lalala') > >>> If there's no guessing and all data is local, how hard can it >>> be? I'm not asking anyone to do it, but it does look like it would be >>> a simpler, more immediate solution. >> >> istm that it can be complex to have it working over the mirrors, >> unless it's a proxy on client-side. >> But then we're closer to a client-side change I guess. > > Given the existing mirrors infrastructure it is indeed hard. If the > mirrors were demoted to just serving downloads and not the index, they > could still be used with such an index as-is. Yeah, the mirror is standalone or it breaks its intent (being useful when PyPI is down) > As I said, I have to > agree it would be impractical to add a stable-only pypi index and try > to mirror it as well. > > > > -- > http://self.maluke.com/ > -- Tarek Ziad? | http://ziade.org From sergey at maluke.com Mon Jul 11 21:56:44 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Mon, 11 Jul 2011 22:56:44 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> Message-ID: <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> On 11 July 2011 22:47, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > On Mon, Jul 11, 2011 at 9:32 PM, Sergey Schetinin <sergey at maluke.com> wrote: >> On 11 July 2011 22:20, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >>> On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: >>> ... >>>> >>>> That webapp (it's not *really* a proxy) took me about two hours total, >>>> and it has to query pypi, guess what is stable and not, cache the >>>> results. >>> >>> how do you guess it ? that's an interesting part >> >> Well, I mentioned it in linked post. Basically at first I was doing just this: >> >> _rx_final = re.compile(r'[\d\.]+\Z') >> >> def is_final(ver): >> ? ?return bool(_rx_final.match(ver)) >> >> But to implement the most-stable filter, I changed it to this: >> http://pastebin.com/JVVAJ9hE > > there must be a bug because all the versions I am trying are marked stable > (even 'lalala') Not a bug. It marks the most stable versions available: >>> mark_stable(['lala']) [('lala', True)] >>> mark_stable(['lala', '1.0']) [('lala', False), ('1.0', True)] >>> mark_stable(['0.1a1']) [('0.1a1', True)] >>> mark_stable(['0.1a1', '0.1b1']) [('0.1a1', False), ('0.1b1', True)] >>> mark_stable(['0.1a1', '0.1b1', '0.1']) [('0.1a1', False), ('0.1b1', False), ('0.1', True)] >>>> If there's no guessing and all data is local, how hard can it >>>> be? I'm not asking anyone to do it, but it does look like it would be >>>> a simpler, more immediate solution. >>> >>> istm that it can be complex to have it working over the mirrors, >>> unless it's a proxy on client-side. >>> But then we're closer to a client-side change I guess. >> >> Given the existing mirrors infrastructure it is indeed hard. If the >> mirrors were demoted to just serving downloads and not the index, they >> could still be used with such an index as-is. > > Yeah, the mirror is standalone or it breaks its intent (being useful > when PyPI is down) It's unfortunate this instability was accepted for granted. -- http://self.maluke.com/ From ziade.tarek at gmail.com Mon Jul 11 22:21:47 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 22:21:47 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> Message-ID: <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> On Mon, Jul 11, 2011 at 9:56 PM, Sergey Schetinin <sergey at maluke.com> wrote: > On 11 July 2011 22:47, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >> On Mon, Jul 11, 2011 at 9:32 PM, Sergey Schetinin <sergey at maluke.com> wrote: >>> On 11 July 2011 22:20, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >>>> On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: >>>> ... >>>>> >>>>> That webapp (it's not *really* a proxy) took me about two hours total, >>>>> and it has to query pypi, guess what is stable and not, cache the >>>>> results. >>>> >>>> how do you guess it ? that's an interesting part >>> >>> Well, I mentioned it in linked post. Basically at first I was doing just this: >>> >>> _rx_final = re.compile(r'[\d\.]+\Z') >>> >>> def is_final(ver): >>> ? ?return bool(_rx_final.match(ver)) >>> >>> But to implement the most-stable filter, I changed it to this: >>> http://pastebin.com/JVVAJ9hE >> >> there must be a bug because all the versions I am trying are marked stable >> (even 'lalala') > > Not a bug. It marks the most stable versions available: > >>>> mark_stable(['lala']) > [('lala', True)] >>>> mark_stable(['lala', '1.0']) > [('lala', False), ('1.0', True)] >>>> mark_stable(['0.1a1']) > [('0.1a1', True)] >>>> mark_stable(['0.1a1', '0.1b1']) > [('0.1a1', False), ('0.1b1', True)] >>>> mark_stable(['0.1a1', '0.1b1', '0.1']) > [('0.1a1', False), ('0.1b1', False), ('0.1', True)] ah ok :D ... >> >> Yeah, the mirror is standalone or it breaks its intent (being useful >> when PyPI is down) > > It's unfortunate this instability was accepted for granted. Yeah, well we've made a lot of progress with PEP 386 / Metadata 1.2 Right now you can still push projects with a version number being "bazinga-boomya". Try to sort this.. PyPI will reject non-pep 386 versions with the newest metadata 1.2, that's why I am pushing hard on its adoption on client-side because it's a better long term solution. Having to guess if a version is stable or not with a version scheme that's free-form is quite a pain. The short-term fix that works well is to ask people not to push unstable stuff at pypi... people can always pick unstable version on the project website Cheers Tarek -- Tarek Ziad? | http://ziade.org From sdouche at gmail.com Tue Jul 12 02:45:36 2011 From: sdouche at gmail.com (Sebastien Douche) Date: Tue, 12 Jul 2011 02:45:36 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> Message-ID: <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> On Mon, Jul 11, 2011 at 22:21, Tarek Ziad? <ziade.tarek at gmail.com> wrote: Hi Tarek > PyPI will reject non-pep 386 versions with the newest metadata 1.2, > that's why I am pushing hard on its adoption on client-side because > it's a better long term solution. Right > Having to guess if a version is stable or not with a version scheme > that's free-form is quite a pain. The short-term fix that works well > is to ask people not to push unstable stuff at pypi... people can > always pick unstable version on the project website Or add a new stable index on PyPI (ex: http://python.org/pypi/stable). Easy to create, easy to maintain, easy to use and don't break anything. -- Sebastien Douche <sdouche at gmail.com> Twitter : @sdouche From martin at v.loewis.de Tue Jul 12 08:14:14 2011 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 12 Jul 2011 08:14:14 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> Message-ID: <4E1BE636.4040904@v.loewis.de> > Or add a new stable index on PyPI (ex: http://python.org/pypi/stable). > Easy to create, easy to maintain, easy to use and don't break > anything. If people can agree on a specification as to what packages this should include exactly, I can happily provide it. I won't lead a project to get people agree, though. If you go by trove classifiers, my proposal would be to include all releases with a classifier "Development Status :: 5 - Production/Stable" or "Development Status :: 6 - Mature". Regards, Martin From sergey at maluke.com Tue Jul 12 08:17:39 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Tue, 12 Jul 2011 09:17:39 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1BE636.4040904@v.loewis.de> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> Message-ID: <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> On 12 July 2011 09:14, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Or add a new stable index on PyPI (ex: http://python.org/pypi/stable). >> Easy to create, easy to maintain, easy to use and don't break >> anything. > > If people can agree on a specification as to what packages this should > include exactly, I can happily provide it. I won't lead a project to > get people agree, though. > > If you go by trove classifiers, my proposal would be to include all > releases with a classifier "Development Status :: 5 - Production/Stable" > or "Development Status :: 6 - Mature". That's very unlikely to work. Once webob got to maturity we set that classifier and it's there even for repo snapshots. I'm sure it's the case for ~100% other projects. -- http://self.maluke.com/ From g.brandl at gmx.net Tue Jul 12 08:28:55 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 12 Jul 2011 08:28:55 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> Message-ID: <ivgpi0$bok$1@dough.gmane.org> Am 12.07.2011 08:17, schrieb Sergey Schetinin: > On 12 July 2011 09:14, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>> Or add a new stable index on PyPI (ex: http://python.org/pypi/stable). >>> Easy to create, easy to maintain, easy to use and don't break >>> anything. >> >> If people can agree on a specification as to what packages this should >> include exactly, I can happily provide it. I won't lead a project to >> get people agree, though. >> >> If you go by trove classifiers, my proposal would be to include all >> releases with a classifier "Development Status :: 5 - Production/Stable" >> or "Development Status :: 6 - Mature". > > That's very unlikely to work. Once webob got to maturity we set that > classifier and it's there even for repo snapshots. I'm sure it's the > case for ~100% other projects. Yeah, the classifier seems to be more appropriate for the project's status, not that of individual releases. (Otherwise it would be a pain to remember to reset that classifier for every prerelease, when all it does is repeat one bit of information already present in the version number.) Georg From ben+python at benfinney.id.au Tue Jul 12 08:37:03 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 12 Jul 2011 16:37:03 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> Message-ID: <87oc10dmy8.fsf@benfinney.id.au> "Martin v. L?wis" <martin at v.loewis.de> writes: > If you go by trove classifiers, my proposal would be to include all > releases with a classifier "Development Status :: 5 - Production/Stable" > or "Development Status :: 6 - Mature". Yes, I think trove classifiers are the right data to use for filtering, and not version string. The ?Development Status? trove category is *exactly* where this information should be found, and corrected if not there. -- \ ?I busted a mirror and got seven years bad luck, but my lawyer | `\ thinks he can get me five.? ?Steven Wright | _o__) | Ben Finney From ben+python at benfinney.id.au Tue Jul 12 08:38:21 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 12 Jul 2011 16:38:21 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> Message-ID: <87k4bodmw2.fsf@benfinney.id.au> Georg Brandl <g.brandl at gmx.net> writes: > (Otherwise it would be a pain to remember to reset that classifier for > every prerelease, when all it does is repeat one bit of information > already present in the version number.) Why is the version string repeating information that is more precisely represented in the trove category? I'd say that's where the problem lies. -- \ ?For every complex problem, there is a solution that is simple, | `\ neat, and wrong.? ?Henry L. Mencken | _o__) | Ben Finney From martin at v.loewis.de Tue Jul 12 08:44:44 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jul 2011 08:44:44 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> Message-ID: <4E1BED5C.3080006@v.loewis.de> > That's very unlikely to work. Once webob got to maturity we set that > classifier and it's there even for repo snapshots. I'm sure it's the > case for ~100% other projects. It seems you are right, although there are certainly counter examples, such as http://pypi.python.org/pypi/zw.schema/0.2.2 http://pypi.python.org/pypi/zw.schema/0.3.0b2.1 In any case, as I said, I'll stay out of any attempt to specify something here. Regards, Martin From sergey at maluke.com Tue Jul 12 08:47:02 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Tue, 12 Jul 2011 09:47:02 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <87oc10dmy8.fsf@benfinney.id.au> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <87oc10dmy8.fsf@benfinney.id.au> Message-ID: <CAP2iL+DpYce3zo_2zvEUnmwpiX6yE24oKT5MMA4wfiD9pVSxYw@mail.gmail.com> On 12 July 2011 09:37, Ben Finney <ben+python at benfinney.id.au> wrote: > "Martin v. L?wis" <martin at v.loewis.de> writes: > >> If you go by trove classifiers, my proposal would be to include all >> releases with a classifier "Development Status :: 5 - Production/Stable" >> or "Development Status :: 6 - Mature". > > Yes, I think trove classifiers are the right data to use for filtering, > and not version string. > > The ?Development Status? trove category is *exactly* where this > information should be found, and corrected if not there. This filtering has to work on as many existing releases as possible, not just the ones made from this point, not even on the most recent ones, it should work with what's actually in pypi. -- http://self.maluke.com/ From ben+python at benfinney.id.au Tue Jul 12 09:12:00 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 12 Jul 2011 17:12:00 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <87oc10dmy8.fsf@benfinney.id.au> <CAP2iL+DpYce3zo_2zvEUnmwpiX6yE24oKT5MMA4wfiD9pVSxYw@mail.gmail.com> Message-ID: <87fwmcdlbz.fsf@benfinney.id.au> Sergey Schetinin <sergey at maluke.com> writes: > On 12 July 2011 09:37, Ben Finney <ben+python at benfinney.id.au> wrote: > > The ?Development Status? trove category is *exactly* where this > > information should be found, and corrected if not there. > > This filtering has to work on as many existing releases as possible, > not just the ones made from this point, not even on the most recent > ones, it should work with what's actually in pypi. Well, if the proposal involves living with bad data and not fixing it, I withdraw my approval (for whatever that was worth). -- \ ?When I was a kid I used to pray every night for a new bicycle. | `\ Then I realised that the Lord doesn't work that way so I stole | _o__) one and asked Him to forgive me.? ?Emo Philips | Ben Finney From g.brandl at gmx.net Tue Jul 12 09:18:38 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 12 Jul 2011 09:18:38 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <87k4bodmw2.fsf@benfinney.id.au> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> <87k4bodmw2.fsf@benfinney.id.au> Message-ID: <ivgsf7$pkl$1@dough.gmane.org> Am 12.07.2011 08:38, schrieb Ben Finney: > Georg Brandl <g.brandl at gmx.net> writes: > >> (Otherwise it would be a pain to remember to reset that classifier for >> every prerelease, when all it does is repeat one bit of information >> already present in the version number.) > > Why is the version string repeating information that is more precisely > represented in the trove category? I'd say that's where the problem > lies. Sorry, I don't understand. Are you suggesting calling both 1.0 alpha 1 and 1.0 final "version 1.0" and just changing the trove classifier? Georg From merwok at netwok.org Tue Jul 12 15:14:22 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 12 Jul 2011 15:14:22 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <ivgpi0$bok$1@dough.gmane.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> Message-ID: <4E1C48AE.4060409@netwok.org> Le 12/07/2011 08:28, Georg Brandl a ?crit : > Am 12.07.2011 08:17, schrieb Sergey Schetinin: >> On 12 July 2011 09:14, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>> If you go by trove classifiers, my proposal would be to include all >>> releases with a classifier "Development Status :: 5 - Production/Stable" >>> or "Development Status :: 6 - Mature". >> >> That's very unlikely to work. Once webob got to maturity we set that >> classifier and it's there even for repo snapshots. I'm sure it's the >> case for ~100% other projects. > > Yeah, the classifier seems to be more appropriate for the project's status, > not that of individual releases. ISTM that the discussion is confusing ?stable? and ?final?. In my understanding, stable = project status = Trove classifier final = release status = PEP 386 version Regards From merwok at netwok.org Tue Jul 12 15:25:34 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 12 Jul 2011 15:25:34 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <ivf9aq$bgv$1@dough.gmane.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> Message-ID: <4E1C4B4E.7030107@netwok.org> Le 11/07/2011 18:45, Tres Seaver a ?crit : > On 07/11/2011 08:41 AM, ?ric Araujo wrote: >> I was under the impression that PEP 386 only defined the syntax of >> version numbers and a comparison algo, but no semantics. IOW there >> is no way for a tool to know that 2.6.33 is devel and 2.6.34 stable, >> or that 1.0.4 does not break compatibility with 1.0.2, or anything >> else of the sort. > > The PEP386 semantics are implied by the sorting order, where "final" > (non-suffixed) releases sort higher than any suffixed releases except > "post" releases. I?m aware of the algo for sorting and asserting final/non-final. I was talking about stability and compatibility policies: PEP 386, contrary to the misnamed Semantic Versioning spec for example, does not define anything in this regard (this was already pointed out a few months ago by PJE, I think). See the examples in the quoted text above. Regards From tseaver at palladion.com Tue Jul 12 15:59:07 2011 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jul 2011 09:59:07 -0400 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1C4B4E.7030107@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> Message-ID: <ivhjvb$9rd$1@dough.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/12/2011 09:25 AM, ?ric Araujo wrote: > Le 11/07/2011 18:45, Tres Seaver a ?crit : >> On 07/11/2011 08:41 AM, ?ric Araujo wrote: >>> I was under the impression that PEP 386 only defined the syntax >>> of version numbers and a comparison algo, but no semantics. IOW >>> there is no way for a tool to know that 2.6.33 is devel and >>> 2.6.34 stable, or that 1.0.4 does not break compatibility with >>> 1.0.2, or anything else of the sort. >> >> The PEP386 semantics are implied by the sorting order, where >> "final" (non-suffixed) releases sort higher than any suffixed >> releases except "post" releases. > > I?m aware of the algo for sorting and asserting final/non-final. I > was talking about stability and compatibility policies: PEP 386, > contrary to the misnamed Semantic Versioning spec for example, does > not define anything in this regard (this was already pointed out a > few months ago by PJE, I think). See the examples in the quoted > text above. Hmm? In what sense would 2.6.33 ever be used for a "devel" release? If my application got broken by a project that made such a release, I would be busy ripping out a dependency produced by such an unreliable project, not arguing whether PyPI could / should implement some "technical measure" to make everything happy. If your point is that a "stable-only" mirror could still induce breakage on projects which use it blindly, I certainly concur. In fact, I have long asserted that any "integrator" whose production builds uses PyPI directly is solely responsible for the breakage, when (not if) it occurs. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4cUysACgkQ+gerLs4ltQ5kQwCg1Ed7iv0/1ag1NxN07vqCVWbl IAsAoM/1xY5BPu/4AG0aCQdxLhQnEQEh =I+yw -----END PGP SIGNATURE----- From merwok at netwok.org Tue Jul 12 16:26:45 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 12 Jul 2011 16:26:45 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <ivhjvb$9rd$1@dough.gmane.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> Message-ID: <4E1C59A5.4020406@netwok.org> > Hmm? In what sense would 2.6.33 ever be used for a "devel" release? By the linux kernel project, for example: http://en.wikipedia.org/wiki/Software_versioning#Odd-numbered_versions_for_development_releases > If my application got broken by a project that made such a release, I > would be busy ripping out a dependency produced by such an unreliable > project, not arguing whether PyPI could / should implement some > "technical measure" to make everything happy. Granted, this linux kernel versioning example was extreme. However, compatibility issues between X.Y.Z and X.Y.Z+1 is not unheard of, and that?s a problem that PEP 386 does not address. (Even if it did, nothing would prevent human error or malice, we?d still need human checks, but at least we?d have a common base for expectations.) > If your point is that a "stable-only" mirror could still induce breakage > on projects which use it blindly, I certainly concur. My point is that ?stability? is not defined, at least not by PEP 386. And even when the Trove classifiers are set, blindly trusting them would not be a good idea. Regards From ziade.tarek at gmail.com Tue Jul 12 16:44:57 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 12 Jul 2011 16:44:57 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1C59A5.4020406@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> <4E1C59A5.4020406@netwok.org> Message-ID: <CAGSi+Q54-M9vXDFTedbwaHR2VW9F8jR9wfcPR-yiom4qgcfHiA@mail.gmail.com> On Tue, Jul 12, 2011 at 4:26 PM, ?ric Araujo <merwok at netwok.org> wrote: >> Hmm? ?In what sense would 2.6.33 ever be used for a "devel" release? > > By the linux kernel project, for example: > http://en.wikipedia.org/wiki/Software_versioning#Odd-numbered_versions_for_development_releases > >> If my application got broken by a project that made such a release, I >> would be busy ripping out a dependency produced by such an unreliable >> project, not arguing whether PyPI could / should implement some >> "technical measure" to make everything happy. > > Granted, this linux kernel versioning example was extreme. ?However, > compatibility issues between X.Y.Z and X.Y.Z+1 is not unheard of, and > that?s a problem that PEP 386 does not address. ?(Even if it did, > nothing would prevent human error or malice, we?d still need human > checks, but at least we?d have a common base for expectations.) We provide a versionning scheme with a dev tag to be used for dev releases, if people push a dev version under a version number without the dev flag, that's their fault. There's no difference here between a trove classifier and a version scheme. Those are just conventions, flags. So I am not sure how you want to address this issue exactly, a dev marker is a dev marker... and dev/non-dev != stability for backward compat issues, it's a problem to be addressed at the project level, with its own conventions. > >> If your point is that a "stable-only" mirror could still induce breakage >> on projects which use it blindly, I certainly concur. > > My point is that ?stability? is not defined, at least not by PEP 386. You can't define what stability means in a generic way in the versionning scheme. We have dev marker, beta alpa, rc markers but that's just transitional states between dev and prod for the sake of feedback. It's up to every project to define stability, and I agree that the Trove is a good way to do it. > And even when the Trove classifiers are set, blindly trusting them would > not be a good idea. Ah you lost me here. If the project uses a trove classifier to mark "stable", either you trust the project either you don't. Deciding if you upgrade to the newer version in your environment is your own problem and orthogonal > > Regards > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- Tarek Ziad? | http://ziade.org From tseaver at palladion.com Tue Jul 12 16:55:31 2011 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jul 2011 10:55:31 -0400 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1C59A5.4020406@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> <4E1C59A5.4020406@netwok.org> Message-ID: <ivhn93$u7k$1@dough.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/12/2011 10:26 AM, ?ric Araujo wrote: >> Hmm? In what sense would 2.6.33 ever be used for a "devel" >> release? > > By the linux kernel project, for example: > http://en.wikipedia.org/wiki/Software_versioning#Odd-numbered_versions_for_development_releases That > would have been 2.3.36 vs. 2.2 29: the even / odd variation for development / stable was based on the minor release number, not the third. Also note that the kernel devs dropped that pattern quite a while back as being unfruitful. >> If my application got broken by a project that made such a release, >> I would be busy ripping out a dependency produced by such an >> unreliable project, not arguing whether PyPI could / should >> implement some "technical measure" to make everything happy. > > Granted, this linux kernel versioning example was extreme. However, > compatibility issues between X.Y.Z and X.Y.Z+1 is not unheard of, > and that?s a problem that PEP 386 does not address. (Even if it > did, nothing would prevent human error or malice, we?d still need > human checks, but at least we?d have a common base for > expectations.) We *do* have a common base for expectations: the set of projects on PyPI which violate those expectations (about the semantics of the version number) is pretty small. Backward-[in]compatibility policy is just not a documented part of those expectations: you have to learn each project's policy about BBB issues as part of evaluating whether to use its releases. >> If your point is that a "stable-only" mirror could still induce >> breakage on projects which use it blindly, I certainly concur. > > My point is that ?stability? is not defined, at least not by PEP > 386. And even when the Trove classifiers are set, blindly trusting > them would not be a good idea. Sure: blindly trusting any project / author is not a good idea for people who need highly-reliable / reproducible build-deployment solutions. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4cYGMACgkQ+gerLs4ltQ6ThwCgn4j9GAf2Vm9TR8C9AQO6VLZ+ bE8AoLQsfXMYK0r3buANjMqegAbl8Lkt =xr3/ -----END PGP SIGNATURE----- From g.brandl at gmx.net Tue Jul 12 20:00:44 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 12 Jul 2011 20:00:44 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1C48AE.4060409@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> <4E1C48AE.4060409@netwok.org> Message-ID: <ivi234$6qf$2@dough.gmane.org> Am 12.07.2011 15:14, schrieb ?ric Araujo: > Le 12/07/2011 08:28, Georg Brandl a ?crit : >> Am 12.07.2011 08:17, schrieb Sergey Schetinin: >>> On 12 July 2011 09:14, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>>> If you go by trove classifiers, my proposal would be to include all >>>> releases with a classifier "Development Status :: 5 - Production/Stable" >>>> or "Development Status :: 6 - Mature". >>> >>> That's very unlikely to work. Once webob got to maturity we set that >>> classifier and it's there even for repo snapshots. I'm sure it's the >>> case for ~100% other projects. >> >> Yeah, the classifier seems to be more appropriate for the project's status, >> not that of individual releases. > > ISTM that the discussion is confusing ?stable? and ?final?. > > In my understanding, > stable = project status = Trove classifier > final = release status = PEP 386 version Exactly. While a project itself can be stable (or mature, even), individual releases still can be of pre-release quality. Georg From ben+python at benfinney.id.au Tue Jul 12 23:43:46 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 13 Jul 2011 07:43:46 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> <4E1C48AE.4060409@netwok.org> Message-ID: <877h7ndvjh.fsf@benfinney.id.au> ?ric Araujo <merwok at netwok.org> writes: > ISTM that the discussion is confusing ?stable? and ?final?. > > In my understanding, > stable = project status = Trove classifier You're right, I'm wrong. The trove classifier is more appropriate for the project status. -- \ ?We can't depend for the long run on distinguishing one | `\ bitstream from another in order to figure out which rules | _o__) apply.? ?Eben Moglen, _Anarchism Triumphant_, 1999 | Ben Finney From ben+python at benfinney.id.au Tue Jul 12 23:55:52 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 13 Jul 2011 07:55:52 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> Message-ID: <87zkkjcgev.fsf@benfinney.id.au> Tres Seaver <tseaver at palladion.com> writes: > On 07/12/2011 09:25 AM, ?ric Araujo wrote: > >> On 07/11/2011 08:41 AM, ?ric Araujo wrote: > >>> I was under the impression that PEP 386 only defined the syntax of > >>> version numbers and a comparison algo, but no semantics. IOW there > >>> is no way for a tool to know that 2.6.33 is devel and 2.6.34 > >>> stable, or that 1.0.4 does not break compatibility with 1.0.2, or > >>> anything else of the sort. [?] > > Hmm? In what sense would 2.6.33 ever be used for a "devel" release? Why should it not be? Projects are not bound to insert words anywhere in their version strings for any state of the code. I find inserting words into a version string to be ugly and overly complicated, and I don't recommend it for any project. Fortunately, conforming with PEP 386 doesn't require anyone to do that. Nor does PEP 386 define what a version string like ?2.6.33? means. It only says how that version string will sort against other version strings. > If my application got broken by a project that made such a release, I > would be busy ripping out a dependency produced by such an unreliable > project, not arguing whether PyPI could / should implement some > "technical measure" to make everything happy. That expectation (that a development version must have a particular word inserted in the version string) is unreasonable and has no foundation in PEP 386. A tool for automatically producing a ?stable-only? mirror can't rely on the version string containing any information about the development status of a version; not even one which conforms perfectly to PEP 386. > If your point is that a "stable-only" mirror could still induce breakage > on projects which use it blindly, I certainly concur. In fact, I have > long asserted that any "integrator" whose production builds uses PyPI > directly is solely responsible for the breakage, when (not if) it occurs. +1. This is what OS distributions are good for: selecting which package versions are suitable for the stability of the OS as a whole. -- \ ?I stayed up all night playing poker with tarot cards. I got a | `\ full house and four people died.? ?Steven Wright | _o__) | Ben Finney -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110713/87ac8f34/attachment.pgp> From ben+python at benfinney.id.au Wed Jul 13 00:02:40 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 13 Jul 2011 08:02:40 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> <4E1C59A5.4020406@netwok.org> <CAGSi+Q54-M9vXDFTedbwaHR2VW9F8jR9wfcPR-yiom4qgcfHiA@mail.gmail.com> Message-ID: <87vcv7cg3j.fsf@benfinney.id.au> Tarek Ziad? <ziade.tarek at gmail.com> writes: > We provide a versionning scheme with a dev tag to be used for dev > releases, if people push a dev version under a version number without > the dev flag, that's their fault. How is it a ?fault?? It's up to the project to decide the semantics of their version strings. If PEP 386 is going to be used as a blunt instrument to force semantics (and I agree with Eric that its poorly-chosen name encourages this), then the fault of that is with those who misunderstand what PEP 386 requires. > for backward compat issues, it's a problem to be addressed at the > project level, with its own conventions. Yes, thank you. Let's stop saying ?fault? if a project's chosen meaning for version strings doesn't require words in version strings. > You can't define what stability means in a generic way in the > versionning scheme. We have dev marker, beta alpa, rc markers but > that's just transitional states between dev and prod for the sake of > feedback. It's also not at all required under PEP 386 for any project to use those markers. > It's up to every project to define stability, and I agree that the > Trove is a good way to do it. Yet there seems to be no Trove classifier for the stability of an individual version, as contrasted with the ?overall stability? of the project. Should we re-purpose that trove classifier to this meaning? Maybe. But how to convince everyone to use it that way? -- \ ?First things first, but not necessarily in that order.? ?The | `\ Doctor, _Doctor Who_ | _o__) | Ben Finney From martin at v.loewis.de Thu Jul 14 21:17:05 2011 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 14 Jul 2011 21:17:05 +0200 Subject: [Catalog-sig] b.pypi up-to-date again Message-ID: <4E1F40B1.9030704@v.loewis.de> I fixed the AppEngine PyPI mirror (b.pypi.python.org), and it's up-to-date again. In case you are interested, here is the list of problems: - integration of download stats broke on 2010-01-17. The implementation creates a Download record every time somebody downloads a file from the mirror, and integrates them once a day into a daily report (deleting the Download objects, and creating a Stats object). It combines all Download records except for the ones of the current day, and deletes the one it combines (in chunks of 100, since integrating all at once would exceed the 30s limit in appengine). Now, for Jan 17, integration completed (i.e. GAE said there weren't any further Download objects left); however, the next day, another Download object for the day showed up, breaking the integration job. I fixed it by post-dating it to the next day that hadn't been integrated. - mirroring broke when a file was deleted on PyPI after the mirror learned of its existence, but before it got mirrored. Mirroring works the way that a) the GAE app asks PyPI to upload something (downloading from the master doesn't work because it exceeds the maximum response size for the url fetcher). b) PyPI asks GAE to create an upload URL, and asks what file to upload c) PyPI uploads the file to the blob store d) GAE tells the GAE app that the upload is complete In the scenario, step c) failed, because the file wasn't there anymore, so PyPI just ignored it. The GAE app's cron job then restarted the upload, which would fail again - and so every 5 minutes since March 2010. I fixed it by having PyPI upload a null-byte file, along with a POST flag telling that the file is to be deleted - restarting the mirroring then failed since accessing the changelog since March took about 20s, exceeding the 5s XML-RPC limit of GAE. I fixed it by introducing another RPC, changed_packages. It then took three days to catch up, but should now stay up-to-date. Regards, Martin From lac at openend.se Thu Jul 14 21:24:53 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 14 Jul 2011 21:24:53 +0200 Subject: [Catalog-sig] b.pypi up-to-date again In-Reply-To: Message from =?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?= <martin@v.loewis.de> of "Thu, 14 Jul 2011 21:17:05 +0200." <4E1F40B1.9030704@v.loewis.de> References: <4E1F40B1.9030704@v.loewis.de> Message-ID: <201107141924.p6EJOrC9022351@theraft.openend.se> In a message of Thu, 14 Jul 2011 21:17:05 +0200, "Martin v. L?wis" writes: >I fixed the AppEngine PyPI mirror (b.pypi.python.org), and it's >up-to-date again. Hurrah! (And thank you.) Laura From tjreedy at udel.edu Thu Jul 14 22:25:11 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 14 Jul 2011 16:25:11 -0400 Subject: [Catalog-sig] b.pypi up-to-date again In-Reply-To: <4E1F40B1.9030704@v.loewis.de> References: <4E1F40B1.9030704@v.loewis.de> Message-ID: <ivnjb7$jka$1@dough.gmane.org> On 7/14/2011 3:17 PM, "Martin v. L?wis" wrote: > In case you are interested, here is the list of problems: Thank you for this also. I learn from such reports. -- Terry Jan Reedy From richard at python.org Fri Jul 15 08:41:13 2011 From: richard at python.org (Richard Jones) Date: Fri, 15 Jul 2011 16:41:13 +1000 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <4E175900.6070907@voidspace.org.uk> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> Message-ID: <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> Sorry for the silence. Currently there's no pypi admins alias, though I think it'd probably be nice to have rather than to have three email addresses hard-coded into the application like we have at the moment :-) On the other hand, in terms of this error page - is it possible to customise it to forward the user to the support tracker (ie. the one in the pypi sidebar) rather than email? Richard On 9 July 2011 05:22, Michael Foord <michael at voidspace.org.uk> wrote: > Would it be possible to configure pypi to not recommend contacting > webmaster at python.org on server error, but contact the pypi admins instead? > > All the best, > > Michael Foord > > -------- Original Message -------- Subject: Recieved following error at > 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ Date: Fri, 8 Jul 2011 > 15:11:09 -0400 From: Jordan Goldstein <jordango at mit.edu><jordango at mit.edu> To: > webmaster at python.org > > Enjoy. > > Please try again later. > > The server encountered an internal error or misconfiguration and was unable > to complete your request. > > Please contact the server administrator, webmaster at python.org and inform > them of the time the error occurred, and anything you might have done that > may have caused the error. > > More information about this error may be available in the server error log. > ------------------------------ > Apache/2.2.16 (Debian) Server at pypi.python.org Port 80 > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110715/5005efce/attachment.html> From michael at voidspace.org.uk Fri Jul 15 13:07:00 2011 From: michael at voidspace.org.uk (Michael Foord) Date: Fri, 15 Jul 2011 12:07:00 +0100 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> Message-ID: <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> On 15 July 2011 07:41, Richard Jones <richard at python.org> wrote: > Sorry for the silence. > > Currently there's no pypi admins alias, though I think it'd probably be > nice to have rather than to have three email addresses hard-coded into the > application like we have at the moment :-) > > On the other hand, in terms of this error page - is it possible to > customise it to forward the user to the support tracker (ie. the one in the > pypi sidebar) rather than email? > > The particular error page is just the standard error 500 page served by Apache - and it is possible to configure Apache to serve a custom one for the pypi subdomain. Not that I know *specifically how* to do that, but I know it is possible. :-) A pypi admin address, that just forwards to this list (?) sounds like a very good idea (as well). All the best, Michael > > Richard > > On 9 July 2011 05:22, Michael Foord <michael at voidspace.org.uk> wrote: > >> Would it be possible to configure pypi to not recommend contacting >> webmaster at python.org on server error, but contact the pypi admins >> instead? >> >> All the best, >> >> Michael Foord >> >> -------- Original Message -------- Subject: Recieved following error at >> 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ Date: Fri, 8 Jul 2011 >> 15:11:09 -0400 From: Jordan Goldstein <jordango at mit.edu><jordango at mit.edu> To: >> webmaster at python.org >> >> Enjoy. >> >> Please try again later. >> >> The server encountered an internal error or misconfiguration and was >> unable to complete your request. >> >> Please contact the server administrator, webmaster at python.org and inform >> them of the time the error occurred, and anything you might have done that >> may have caused the error. >> >> More information about this error may be available in the server error >> log. >> ------------------------------ >> Apache/2.2.16 (Debian) Server at pypi.python.org Port 80 >> >> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig >> >> > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110715/6a88c46b/attachment.html> From richard at python.org Fri Jul 15 13:27:40 2011 From: richard at python.org (Richard Jones) Date: Fri, 15 Jul 2011 21:27:40 +1000 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> Message-ID: <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> On 15 July 2011 21:07, Michael Foord <michael at voidspace.org.uk> wrote: > The particular error page is just the standard error 500 page served by > Apache - and it is possible to configure Apache to serve a custom one for > the pypi subdomain. Not that I know *specifically how* to do that, but I > know it is possible. :-) > Yeah, that's pretty much where I'm at :-) Really, the hard part (because I have no control over it) is setting up the alias. > A pypi admin address, that just forwards to this list (?) sounds like a > very good idea (as well). > I'm not sure this list really wants to be caught up in the administrivia of running pypi :-) Richard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110715/e6c8cec4/attachment.html> From fuzzyman at gmail.com Fri Jul 15 13:33:31 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Fri, 15 Jul 2011 12:33:31 +0100 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> Message-ID: <CAKCKLWz7=ATJWQ5Z=4FJQkwCFpw6NTkDe7E9MpHLTUn=xgKPgQ@mail.gmail.com> On 15 July 2011 12:27, Richard Jones <richard at python.org> wrote: > On 15 July 2011 21:07, Michael Foord <michael at voidspace.org.uk> wrote: > >> The particular error page is just the standard error 500 page served by >> Apache - and it is possible to configure Apache to serve a custom one for >> the pypi subdomain. Not that I know *specifically how* to do that, but I >> know it is possible. :-) >> > > Yeah, that's pretty much where I'm at :-) > > Really, the hard part (because I have no control over it) is setting up the > alias. > > You could ask on pydotorg-www, there are a few admins on there. > > >> A pypi admin address, that just forwards to this list (?) sounds like a >> very good idea (as well). >> > > I'm not sure this list really wants to be caught up in the administrivia of > running pypi :-) > > Isn't it already? Another need we have at webmaster at python.org is somewhere to redirect reports of pypi problems. We often get direct email reports of issues. An email alias I could just send forward those emails to would be very handy. Currently I'm using this list (or telling people to report stuff on the tracker - which is a bit unfriendly for someone who has already taken the time to report an issue). All the best, Michael > > Richard > > >> >> > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110715/fdcb3d31/attachment.html> From mal at egenix.com Fri Jul 15 13:45:22 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Jul 2011 13:45:22 +0200 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> Message-ID: <4E202852.9010609@egenix.com> Richard Jones wrote: > On 15 July 2011 21:07, Michael Foord <michael at voidspace.org.uk> wrote: > >> The particular error page is just the standard error 500 page served by >> Apache - and it is possible to configure Apache to serve a custom one for >> the pypi subdomain. Not that I know *specifically how* to do that, but I >> know it is possible. :-) >> > > Yeah, that's pretty much where I'm at :-) > > Really, the hard part (because I have no control over it) is setting up the > alias. You just need to contact postmaster at python.org. They can set it up for you. >> A pypi admin address, that just forwards to this list (?) sounds like a >> very good idea (as well). >> > > I'm not sure this list really wants to be caught up in the administrivia of > running pypi :-) > > > Richard > > >> >> > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From faassen at startifact.com Fri Jul 15 14:04:34 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 14:04:34 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAHrZfZB+DHR1LQmFmsgCyD0_b21hyQRdmDsG8YFiKTrcqwqMDw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> <4E135C13.3010207@v.loewis.de> <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> <CAHrZfZB+DHR1LQmFmsgCyD0_b21hyQRdmDsG8YFiKTrcqwqMDw@mail.gmail.com> Message-ID: <ivpac6$kv3$2@dough.gmane.org> On 07/05/2011 11:55 PM, Richard Jones wrote: > Could we rewind right back to the start of this conversation when it > was suggested that you keep and ship a local mirror of packages you > rely on? Such mirroring is trivial to set up. Why force everybody who depends on PyPI making a local mirror? It may be trivial, but trivial * 1000 is still a lot of work. So I'd like to share the burden of setting it up and maintaining it with others. Regards, Martijn From faassen at startifact.com Fri Jul 15 14:01:18 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 14:01:18 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110706011938.0B6C13A4108@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.g mail.com> <20110706011938.0B6C13A4108@sparrow.telecommunity.com> Message-ID: <ivpa62$kv3$1@dough.gmane.org> On 07/06/2011 03:19 AM, P.J. Eby wrote: > At 08:56 PM 7/5/2011 +0200, Martijn Faassen wrote: >> Sure. Unacceptable interference with the freedom of package authors to >> remove stuff. Unacceptable because developers need >> the freedom to remove old versions of their software from the >> internet. :) > > You still seem to be ignoring the part where insisting that people keep > old things around for your convenience *also* means they get support > requests from everyone *else* on the internet. In other words, a package > author has to pay an (ongoing and perpetual) support cost for your > temporary convenience. > > You are not making a principled moral stand here. Rather, you are asking > that other people be made to carry an additional cost, for your benefit, > on the basis that you don't think their cost is important, but you think > your benefit is. A package once released to the Internet can be kept by someone else on the Internet, no matter what the original package author has to say, if that package has a license which permits it. I'm looking for a way to make packages stay unchanged in a way convenient for myself and others. Regards, Martijn P.S. Concerning morality, way off topic. Perhaps reflect on impugning someone's morality as an aspect of moral behavior? Thanks. From benji at benjiyork.com Fri Jul 15 14:12:03 2011 From: benji at benjiyork.com (Benji York) Date: Fri, 15 Jul 2011 08:12:03 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <ivpac6$kv3$2@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> <4E135C13.3010207@v.loewis.de> <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> <CAHrZfZB+DHR1LQmFmsgCyD0_b21hyQRdmDsG8YFiKTrcqwqMDw@mail.gmail.com> <ivpac6$kv3$2@dough.gmane.org> Message-ID: <CAC8tyQ55+KOOms3HDeZ7eOcZ41hTdZ9KPAm8J5UfXR0+cZAM5w@mail.gmail.com> On Fri, Jul 15, 2011 at 8:04 AM, Martijn Faassen <faassen at startifact.com> wrote: > On 07/05/2011 11:55 PM, Richard Jones wrote: >> >> Could we rewind right back to the start of this conversation when it >> was suggested that you keep and ship a local mirror of packages you >> rely on? Such mirroring is trivial to set up. > > Why force everybody who depends on PyPI making a local mirror? It may be > trivial, but trivial * 1000 is still a lot of work. > > So I'd like to share the burden of setting it up and maintaining it with > others. Indeed. There are enough people that desire a "write only" PyPI that it would be less work globally for those of us that want one to share the burden of building it. On the other hand, even though I'm in the don't-delete-things-from-PyPI camp myself, I'm not sure enough of its users and maintainers share the sentiment that we'll be able to agree to change PyPI to fulfill the roll. We may have to build it outside of PyPI proper. -- Benji York From richard at python.org Fri Jul 15 14:24:53 2011 From: richard at python.org (Richard Jones) Date: Fri, 15 Jul 2011 22:24:53 +1000 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAKCKLWz7=ATJWQ5Z=4FJQkwCFpw6NTkDe7E9MpHLTUn=xgKPgQ@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> <CAKCKLWz7=ATJWQ5Z=4FJQkwCFpw6NTkDe7E9MpHLTUn=xgKPgQ@mail.gmail.com> Message-ID: <CAHrZfZBBnh-aBhT=bpQq4104ZdLhcNK6M4nV+n6rG-2r+AX4pQ@mail.gmail.com> On 15 July 2011 21:33, Michael Foord <fuzzyman at gmail.com> wrote: > > > On 15 July 2011 12:27, Richard Jones <richard at python.org> wrote: >> >> On 15 July 2011 21:07, Michael Foord <michael at voidspace.org.uk> wrote: >>> >>> The particular error page is just the standard error 500 page served by >>> Apache - and it is possible to configure Apache to serve a custom one for >>> the pypi subdomain. Not that I know *specifically how* to do that, but I >>> know it is possible. :-) >> >> Yeah, that's pretty much where I'm at :-) >> Really, the hard part (because I have no control over it) is setting up >> the alias. > > You could ask on pydotorg-www, there are a few admins on there. Indeed that was my next step, but I've been too busy playing Just Cause 2 and trying to download Kirbal Space Program to get around to it :-) Richard From jim at zope.com Fri Jul 15 15:24:54 2011 From: jim at zope.com (Jim Fulton) Date: Fri, 15 Jul 2011 09:24:54 -0400 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) Message-ID: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> On Fri, Jul 15, 2011 at 8:12 AM, Benji York <benji at benjiyork.com> wrote: > On Fri, Jul 15, 2011 at 8:04 AM, Martijn Faassen <faassen at startifact.com> wrote: >> On 07/05/2011 11:55 PM, Richard Jones wrote: >>> >>> Could we rewind right back to the start of this conversation when it >>> was suggested that you keep and ship a local mirror of packages you >>> rely on? Such mirroring is trivial to set up. >> >> Why force everybody who depends on PyPI making a local mirror? It may be >> trivial, but trivial * 1000 is still a lot of work. >> >> So I'd like to share the burden of setting it up and maintaining it with >> others. > > Indeed. ?There are enough people that desire a "write only" PyPI that > it would be less work globally for those of us that want one to share > the burden of building it. > > On the other hand, even though I'm in the > don't-delete-things-from-PyPI camp myself, I'm not sure enough of its > users and maintainers share the sentiment that we'll be able to agree > to change PyPI to fulfill the roll. ?We may have to build it outside > of PyPI proper. You may be right, however: 1. I think if package maintainers actually knew that removing old revisions did harm, most would not delete revisions or packages unless they thought it was necessary. I think most people who remove old revisions do so for cleanliness, not appreciating that there is a downside. 2. I think a lot of people were turned off by the mandatory nature of the original proposal. (I now regret my +1 of that proposal, made in support of the sentiment behind it, but not taking into consideration how it runs counter to Python culture.) PROPOSAL ========= I propose we *try* simply adding an informative confirmation when removing a package or revision that simply explains that removal may inconvenience people relying on the package or the specific version. I predict that there will be a lot fewer annoying removals of we do this. I'd also like to remind everyone that on rare occasions, there are good reasons to remove a package or revision and the sort of mirror contemplated here would make that a lot harder. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From benji at benjiyork.com Fri Jul 15 15:43:47 2011 From: benji at benjiyork.com (Benji York) Date: Fri, 15 Jul 2011 09:43:47 -0400 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> Message-ID: <CAC8tyQ6oBXmGr7OBbAvk_+oqSoZStKQjN7B8uUDa1nk5Tte0Qg@mail.gmail.com> On Fri, Jul 15, 2011 at 9:24 AM, Jim Fulton <jim at zope.com> wrote: > 1. I think if package maintainers actually knew that removing old > revisions did harm, most would not delete revisions or packages unless > they thought it was necessary. ?I think most people who remove old > revisions do so for cleanliness, not appreciating that there is a > downside. > PROPOSAL > ========= > > I propose we *try* simply adding an informative confirmation when > removing a package or revision that simply explains that removal may > inconvenience people relying on the package or the specific version. > I predict that there will be a lot fewer annoying removals of we do > this. +1 This is cheap to do and doesn't restrict a package owner's ability to remove a package if they really feel the need. It'd be really nice if we could quantify the number of packages removed before and after the change, but I'm not sure we collect enough data to do so. -- Benji York From faassen at startifact.com Fri Jul 15 16:36:28 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 16:36:28 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E14851C.6070402@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> Message-ID: <ivpj90$f4e$1@dough.gmane.org> Hi there, On 07/06/2011 05:54 PM, M.-A. Lemburg wrote: >> This is not a good reason, IMHO. You can go on with new versions and a >> new name, maybe you could want to deprecate the old package, but it's >> not a good reason to remove it. > > I think undoing mistakes in package names is a very > good reason to remove them. As package author you don't want such > mistakes to stay on the net forever, if you can avoid it. I don't understand the "you don't want such mistakes to stay on the net forever" line of argumentation. As a package author, once you release something to the internet, you can assume it'll stay on the net. This is already true. So this is an unfixable problem that we're not making worse by having an immutable mirror. >>> * legal action (copyright, trademark, DMCA, license issues, etc) >>> >>> * removal of malicious packages (e.g. script kiddy stuff in >>> setup.py) >>> * seriously broken builds (e.g. that cause users to lose data) >> >> This is would make a good reason for package removal, but not for >> version reassignment. I.E. if I delete version 0.4.3 because it >> deleted my /usr instead of /usr/share/mylib/content, then I would be >> right at removing it, but there's no point in allowing any other >> package to take back 0.4.3. It's simply gone. > > I'm not sure I understand what you want to say. I wasn't talking > about version reassignment in the above cases. Well, it restricts the use cases. The use case "re-release Foo 3.0 but with different contents" doesn't seem to be necessary to tackle the above, just plain removal. So a friendly perpetual mirror would need to be able to handle delete requests but not re-upload requests. >>> * reassigning package names (not sure whether that's possible with >>> PyPI, but it certainly happens in the wild every now and then) >> >> I'm not sure about what you mean here. > > Author A releases a package X, then drops the idea and removes > the package, freeing up the name for others to use. Later on, > author B uses the name X for something different and creates > a new package X with a new set of releases. I wonder by the way whether PyPI supports the "dropping package name forever" use case now. I think a lot is to be gained if you assume package names to be unique forever. For instance: allowing this would be a major security risk: I release package X. Package X is used by people. Then I drop the name. Someone else completely unrelated comes along, creates package with the same name, and uploads evil code. Hilarity ensues. I know the answer already: "Just don't use packages by people who will drop the name at a nebulous point in the future then!" :) Yes, allowing this follows from the "developers should have total freedom" goal that is apparently the main driver behind PyPI's use cases, but there are also "security please" and "repeatability" use cases. Regards, Martijn From faassen at startifact.com Fri Jul 15 16:55:47 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 16:55:47 +0200 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> Message-ID: <ivpkd6$mv1$1@dough.gmane.org> On 07/15/2011 03:24 PM, Jim Fulton wrote: [snip] > 1. I think if package maintainers actually knew that removing old > revisions did harm, most would not delete revisions or packages unless > they thought it was necessary. I think most people who remove old > revisions do so for cleanliness, not appreciating that there is a > downside. > > 2. I think a lot of people were turned off by the mandatory nature of > the original proposal. (I now regret my +1 of that proposal, made in > support of the sentiment behind it, but not taking into consideration > how it runs counter to Python culture.) [snip proposal] Sure, I already indicated a +1 earlier. I tried to sketch out the use cases behind my proposal but people focused on what turned them off. So thinking about this more, what about an extra feature that focuses on communication? What I'm interested in is the following: * reducing the incidence of suddenly removed packages (the message might help) * being aware that packages are removed Others also expressed an interest in having the following: * having a way to deprecate old releases or whole packages So what if we provided communication channels for that? if a package or release is removed by an author, they could be asked to give a reason ("botched upload!", "legal reasons"), whatever. Then there'll be a feed which shows recent deletions along with the reasons. There'd be some machine readable components in this feed too. Also useful for tools would be way to access the total list of all deletions. This could also be expanded to a "deprecation message". Instead of removing a package, an author can add a deprecation message to a package. This could also be on a feed. This way we have a communication channel where we can be informed about what's going on. In response both humans as well as automated tools could take action. You could for instance have a scanning tool which looks at installed packages and tells you which ones are deprecated or removed. It doesn't offer the same guarantees as a perpetual mirror would, so it doesn't quite solve the same use cases. But it would make it possible for people to properly collaborate and communicate about this kind of thing. I'm not proposing that any of the existing client-side tools are adjusted to support any of this. tldr: we've got a feed and index for packages. what about a feed and index for removals and deprecations? Regards, Martijn From g.brandl at gmx.net Fri Jul 15 17:14:09 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 15 Jul 2011 17:14:09 +0200 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> Message-ID: <ivpler$rc5$1@dough.gmane.org> Am 15.07.2011 15:24, schrieb Jim Fulton: > PROPOSAL > ========= > > I propose we *try* simply adding an informative confirmation when > removing a package or revision that simply explains that removal may > inconvenience people relying on the package or the specific version. > I predict that there will be a lot fewer annoying removals of we do > this. That's a very good compromise. +1. Georg From doug.hellmann at gmail.com Fri Jul 15 17:16:57 2011 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Fri, 15 Jul 2011 11:16:57 -0400 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <ivpler$rc5$1@dough.gmane.org> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> <ivpler$rc5$1@dough.gmane.org> Message-ID: <BB8097D4-04AB-4C5C-B6D8-9505DF315B0F@gmail.com> On Jul 15, 2011, at 11:14 AM, Georg Brandl wrote: > Am 15.07.2011 15:24, schrieb Jim Fulton: > >> PROPOSAL >> ========= >> >> I propose we *try* simply adding an informative confirmation when >> removing a package or revision that simply explains that removal may >> inconvenience people relying on the package or the specific version. >> I predict that there will be a lot fewer annoying removals of we do >> this. > > That's a very good compromise. +1. +1 Doug From mal at egenix.com Fri Jul 15 17:59:31 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Jul 2011 17:59:31 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivpj90$f4e$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> Message-ID: <4E2063E3.5080705@egenix.com> Martijn Faassen wrote: > Hi there, > > On 07/06/2011 05:54 PM, M.-A. Lemburg wrote: >>> This is not a good reason, IMHO. You can go on with new versions and a >>> new name, maybe you could want to deprecate the old package, but it's >>> not a good reason to remove it. >> >> I think undoing mistakes in package names is a very >> good reason to remove them. As package author you don't want such >> mistakes to stay on the net forever, if you can avoid it. > > I don't understand the "you don't want such mistakes to stay on the net > forever" line of argumentation. As a package author, once you release > something to the internet, you can assume it'll stay on the net. This is > already true. So this is an unfixable problem that we're not making > worse by having an immutable mirror. It's not unfixable, but it's not easy fo fix either. In any case, we shouldn't be making it harder for authors. We shouldn't forget that we have Python users that are young, not very good at naming things, not aware of consequences that their naming scheme may have, careless, etc. etc. We don't want to make things hard for them, otherwise they'd simply turn away from us, instead of learning how to work with PyPI properly. We do have a responsibility here as community and simply saying: "you uploaded it, so it's your fault, that it'll stay up there", blaming the author forever, does not align well with our Python community spirit. >>>> * legal action (copyright, trademark, DMCA, license issues, etc) >>>> >>>> * removal of malicious packages (e.g. script kiddy stuff in >>>> setup.py) >>>> * seriously broken builds (e.g. that cause users to lose data) >>> >>> This is would make a good reason for package removal, but not for >>> version reassignment. I.E. if I delete version 0.4.3 because it >>> deleted my /usr instead of /usr/share/mylib/content, then I would be >>> right at removing it, but there's no point in allowing any other >>> package to take back 0.4.3. It's simply gone. >> >> I'm not sure I understand what you want to say. I wasn't talking >> about version reassignment in the above cases. > > Well, it restricts the use cases. The use case "re-release Foo 3.0 but > with different contents" doesn't seem to be necessary to tackle the > above, just plain removal. > > So a friendly perpetual mirror would need to be able to handle delete > requests but not re-upload requests. It all depends ... again, it's best to keep the files around that you've tested, rather than relying on some stored on some external file server. >>>> * reassigning package names (not sure whether that's possible with >>>> PyPI, but it certainly happens in the wild every now and then) >>> >>> I'm not sure about what you mean here. >> >> Author A releases a package X, then drops the idea and removes >> the package, freeing up the name for others to use. Later on, >> author B uses the name X for something different and creates >> a new package X with a new set of releases. > > I wonder by the way whether PyPI supports the "dropping package name > forever" use case now. Sure: You file a ticket and Martin or Richard removes the package. > I think a lot is to be gained if you assume package names to be unique > forever. > > For instance: allowing this would be a major security risk: I release > package X. Package X is used by people. Then I drop the name. Someone > else completely unrelated comes along, creates package with the same > name, and uploads evil code. Hilarity ensues. > > I know the answer already: "Just don't use packages by people who will > drop the name at a nebulous point in the future then!" :) As PyPI grows, we'll sooner or later see popular names being reserved on PyPI or names being used for PyPI/distutils/setuptools experiments/learning. Removal of the latter already does occur now, e.g. with the linked list packages described in a Python book for learning distutils packaging. > Yes, allowing this follows from the "developers should have total > freedom" goal that is apparently the main driver behind PyPI's use > cases, but there are also "security please" and "repeatability" use cases. There's no security on PyPI and repeatability is a myth as well :-) Just because we don't have malicious packages on PyPI, doesn't mean it's going to stay like this forever and for repeatability you're better off relying on files you've already downloaded and tested, since it is possible to reupload a package release file with different content, or to change the meta data of a release after its initial upload. I'd suggest you build a read-only PyPI mirror tool that people can download and then use on their private net as they see fit. This solves you use case and that of the other proponents of the read-only PyPI idea, while leaving the mainland PyPI index unchanged. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Fri Jul 15 18:00:05 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Jul 2011 18:00:05 +0200 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <BB8097D4-04AB-4C5C-B6D8-9505DF315B0F@gmail.com> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> <ivpler$rc5$1@dough.gmane.org> <BB8097D4-04AB-4C5C-B6D8-9505DF315B0F@gmail.com> Message-ID: <4E206405.6070103@egenix.com> Doug Hellmann wrote: > > On Jul 15, 2011, at 11:14 AM, Georg Brandl wrote: > >> Am 15.07.2011 15:24, schrieb Jim Fulton: >> >>> PROPOSAL >>> ========= >>> >>> I propose we *try* simply adding an informative confirmation when >>> removing a package or revision that simply explains that removal may >>> inconvenience people relying on the package or the specific version. >>> I predict that there will be a lot fewer annoying removals of we do >>> this. >> >> That's a very good compromise. +1. > > +1 +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From faassen at startifact.com Fri Jul 15 18:20:53 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 18:20:53 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E2063E3.5080705@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> Message-ID: <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> Hi there, On Fri, Jul 15, 2011 at 5:59 PM, M.-A. Lemburg <mal at egenix.com> wrote: >> Yes, allowing this follows from the "developers should have total >> freedom" goal that is apparently the main driver behind PyPI's use >> cases, but there are also "security please" and "repeatability" use cases. > > There's no security on PyPI and repeatability is a myth as well :-) Yes, there is no security and repeatability *now*. That's why I started this whole discussion in the first place, right? You need to argue why these are *not use cases* for PyPI, not that they're not there now. Because it seems if PyPI doesn't care about those, PyPI is really not usable for automated downloading of packages at all. And it seems that is a use case of PyPI? > Just because we don't have malicious packages on PyPI, doesn't > mean it's going to stay like this forever and for repeatability > you're better off relying on files you've already downloaded and > tested, since it is possible to reupload a package release file > with different content, or to change the meta data of a release > after its initial upload. Yes, I talked about reuploading packages and the risks involved in that before, but at least that's not possible by someone else who has nothing to do with the removed package in the first place. Removing a package entirely and letting the name be reused opens up the possibility for completely unrelated people to come in and do nasty things. > I'd suggest you build a read-only PyPI mirror tool that > people can download and then use on their private net as > they see fit. This solves you use case and that of the other > proponents of the read-only PyPI idea, while leaving the > mainland PyPI index unchanged. I will explain again why this doesn't solve my use case. I don't work in a vacuum. I share code with others. This code has dependencies on other code. So how do people obtain this other code? With a private mirror, I'd need to first run all that infrastructure, and then give people access to my private mirror. How private is it going to stay then? How many people are using my code? What maintenance burden does that entail for me? Or I need to give them a possibly giant tarball of all the packages I'm using. Is that really the only way forward for sharing code? Even for libraries with dependencies? PyPI I thought was among other things central place where people can download and install packages from so that they can resolve dependencies, but you seem to be arguing against doing that. At most it's some kind of showcase for packages that peoples should take into their consideration. Taking this point to the extreme, it's *never* something that you can automate downloading from. Instead you should be giving a giant tarball of packages to everybody, always, if they use your code at all. Then this will need to be explained to the people who *are* using PyPI for this purpose. I.e. some significant fraction of all Python programmers out there. Regards, Martijn From martin at v.loewis.de Fri Jul 15 22:58:31 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 15 Jul 2011 22:58:31 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivpj90$f4e$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> Message-ID: <4E20A9F7.1050000@v.loewis.de> >> Author A releases a package X, then drops the idea and removes >> the package, freeing up the name for others to use. Later on, >> author B uses the name X for something different and creates >> a new package X with a new set of releases. > > I wonder by the way whether PyPI supports the "dropping package name > forever" use case now. Of course. When you delete a package, all traces of it are deleted from PyPI for good, except for a log record stating that the package was deleted (and by whom, which again isn't published). Regards, Martin From ben+python at benfinney.id.au Sat Jul 16 01:08:31 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 16 Jul 2011 09:08:31 +1000 Subject: [Catalog-sig] an immutable mirror of PyPI References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> Message-ID: <87aacfb0r4.fsf@benfinney.id.au> Martijn Faassen <faassen at startifact.com> writes: > I don't work in a vacuum. I share code with others. This code has > dependencies on other code. So how do people obtain this other code? By depending on other code, you have a choice to make: you either take the maintenance burden on yourself, or you delegate the maintenance burden (usually to the developers of that code). By delegating the maintenance burden of that code elsewhere, that entails delegating the responsibility for future availability of that code. > PyPI I thought was among other things central place where people can > download and install packages from so that they can resolve > dependencies, but you seem to be arguing against doing that. I find it strange that I'm defending PyPI in this instance, since I am quite sympathetic to complaints that it has poor policies on package availability and many other complaints. But you seem to expect that PyPI must guarantee that any package version ever available will be available forever. That's not reasonable, I think. Instead, you need to choose packages considering whether you trust the package to remain available, which is a social issue between you and the people developing that work. If you think there is a significant risk the people responsible for that package will remove a version on which you depend from PyPI, you should engage in dialogue with those people to resolve that. I don't think PyPI has any business requiring package developers to keep a version available at PyPI beyond when they want it available there. The risks inherent in that need to be addressed as a social issue, not a technical limitation. > At most it's some kind of showcase for packages that peoples should > take into their consideration. Taking this point to the extreme, it's > *never* something that you can automate downloading from. There are points that can be made toward that view; but I don't find this specific case (wanting guaranteed availability of every version forever at PyPI) supports it. > Instead you should be giving a giant tarball of packages to everybody, > always, if they use your code at all. This is indeed a terrible option, and I lament it whenever I see it. I prefer supporting the efforts of those who *do* provide reasonable guarantees of package selection and availability and integration testing. We call them ?operating system distributions?. -- \ ?In general my children refuse to eat anything that hasn't | `\ danced on television.? ?Erma Bombeck | _o__) | Ben Finney From faassen at startifact.com Sat Jul 16 12:54:53 2011 From: faassen at startifact.com (Martijn Faassen) Date: Sat, 16 Jul 2011 12:54:53 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <87aacfb0r4.fsf@benfinney.id.au> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> <87aacfb0r4.fsf@benfinney.id.au> Message-ID: <ivrql8$oq2$1@dough.gmane.org> On 07/16/2011 01:08 AM, Ben Finney wrote: > Martijn Faassen<faassen at startifact.com> writes: > >> I don't work in a vacuum. I share code with others. This code has >> dependencies on other code. So how do people obtain this other code? > > By depending on other code, you have a choice to make: you either take > the maintenance burden on yourself, or you delegate the maintenance > burden (usually to the developers of that code). > > By delegating the maintenance burden of that code elsewhere, that > entails delegating the responsibility for future availability of that > code. There is maintenance burden and there is the package actually existing for download. When I depend on Foo 1.1, I am not delegating maintenance burden to the original developer, unless I go and ask questions about Foo 1.1. The answer can then be: Foo 1.1 is not maintained, sorry. Only when I am interested in upgrades does the original developer come in again. I don't see why these two should be the same: the future availability of an existing release of a package is not identical to continued development of that code. >> PyPI I thought was among other things central place where people can >> download and install packages from so that they can resolve >> dependencies, but you seem to be arguing against doing that. > > I find it strange that I'm defending PyPI in this instance, since I am > quite sympathetic to complaints that it has poor policies on package > availability and many other complaints. > > But you seem to expect that PyPI must guarantee that any package version > ever available will be available forever. That's not reasonable, I > think. I am not barging in here with expectations. I'm coming in here with use cases and proposals. It seems my use cases are rejected as goals of PyPI. In that case I want to get a better understanding of the goals of PyPI. You say that the goal of perpetual availability of packages is an unreasonable goal of PyPI or related services. You don't seem to explain why. So I have use cases: I can release code that relies on releases that can disappear or can be replaced. I think this is bad for repeatability and security. I'd like to see some improvements made. How would we make these improvements? I've so far proposed three ideas: * PyPI not throwing away things after a grace period. Almost universally rejected idea * an additional service, a mirror, that offers some repeatability guarantees. Removal would need to go through channels, implying some kind of custodianship I think people here are wary about. * better communication channels: a list of what's been removed, a list of what's been deprecated. I can then write tools that help me maintain my projects. It's not the same as the above ideas: old projects can still break at the whim of people whose code I depend on, but it'll at least help manage this issue. But perhaps you have better ideas on how to better help manage this. I am getting a bit tired of hearing "you can do this yourself", as this ties into to heart of collaboration, and PyPI if anything is at least supports collaboration. > Instead, you need to choose packages considering whether you trust the > package to remain available, which is a social issue between you and the > people developing that work. > If you think there is a significant risk the people responsible for that > package will remove a version on which you depend from PyPI, you should > engage in dialogue with those people to resolve that. And how exactly am I supposed to read people's minds, possibly years into the future? I had absolutely no expectation that this would happen with the release that disappeared on over a month ago. The developer one day just decided to clean up old, unsupported releases. Of course I contacted the developer after it happened. Several others did too. I then started thinking about how to reduce this risk in a more broad sense. > I don't think PyPI has any business requiring package developers to keep > a version available at PyPI beyond when they want it available there. > The risks inherent in that need to be addressed as a social issue, not a > technical limitation. Yes, this is a social issue. But tools can support social issues. If people tell me to keep my own private mirror, that's a tool solution too, but not a very social one. >> At most it's some kind of showcase for packages that peoples should >> take into their consideration. Taking this point to the extreme, it's >> *never* something that you can automate downloading from. > > There are points that can be made toward that view; but I don't find > this specific case (wanting guaranteed availability of every version > forever at PyPI) supports it. > >> Instead you should be giving a giant tarball of packages to everybody, >> always, if they use your code at all. > > This is indeed a terrible option, and I lament it whenever I see it. > > I prefer supporting the efforts of those who *do* provide reasonable > guarantees of package selection and availability and integration > testing. We call them ?operating system distributions?. The requirements for developers concerning library availability are not identical to those of users. Operating system distributions focus on a stable platform for users. Some developers need to develop cross-platform code. Some developers need to develop different versions of a project, or different project that rely on different versions of dependencies. Some developers need to depend on libraries or library versions not (yet) available in distributions. These developers effectively create a stable distribution of dependencies that they have tested together. It's useful to have tools to support this and allow these developers to share their code with others. Regards, Martijn From faassen at startifact.com Sat Jul 16 12:58:36 2011 From: faassen at startifact.com (Martijn Faassen) Date: Sat, 16 Jul 2011 12:58:36 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E20A9F7.1050000@v.loewis.de> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> Message-ID: <ivrqs7$ppb$1@dough.gmane.org> On 07/15/2011 10:58 PM, "Martin v. L?wis" wrote: >>> Author A releases a package X, then drops the idea and removes >>> the package, freeing up the name for others to use. Later on, >>> author B uses the name X for something different and creates >>> a new package X with a new set of releases. >> >> I wonder by the way whether PyPI supports the "dropping package name >> forever" use case now. > > Of course. When you delete a package, all traces of it are deleted from > PyPI for good, except for a log record stating that the package was > deleted (and by whom, which again isn't published). Okay, so this scenario is possible: * developer of a popular package gets fed up for unknown reasons * removes his package from PyPI (not realizing the thing below) * someone else notices this and recreates the package maliciously * people who download the package (possibly indirectly, by downloading a library that uses this as a dependency) will be bitten It's not an extremely likely scenario but as PyPI grows it becomes possible. I wonder whether there are tooling solutions possible to detect this before it's too late. A public log of what got removed would be useful so people can keep an eye on things - but for this to be caught it would mean that the log would need to include recreations as well. Regards, Martijn From benji at benjiyork.com Sat Jul 16 18:40:16 2011 From: benji at benjiyork.com (Benji York) Date: Sat, 16 Jul 2011 12:40:16 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivrqs7$ppb$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> Message-ID: <CAC8tyQ6Di2Xh4+5EQ_2fLk9MP9rj4J6q8TNfVaB9DMkyLeboVw@mail.gmail.com> On Sat, Jul 16, 2011 at 6:58 AM, Martijn Faassen <faassen at startifact.com> wrote: > I wonder whether there are tooling solutions possible to detect this before > it's too late. A public log of what got removed would be useful so people > can keep an eye on things - but for this to be caught it would mean that the > log would need to include recreations as well. Being a buildout user, if I were to tackle that I'd add something along the lines of SSH's warnings when a host fingerprint changes. I.e., require that package hashes be given (much like you can require that versions be specified) and check those on download. -- Benji York From faassen at startifact.com Sat Jul 16 18:50:07 2011 From: faassen at startifact.com (Martijn Faassen) Date: Sat, 16 Jul 2011 18:50:07 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAC8tyQ6Di2Xh4+5EQ_2fLk9MP9rj4J6q8TNfVaB9DMkyLeboVw@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <CAC8tyQ6Di2Xh4+5EQ_2fLk9MP9rj4J6q8TNfVaB9DMkyLeboVw@mail.gmail.com> Message-ID: <CAGT4ZFgFaXVVr8=e2R81LMZ_YSYtMzcmnroOgvqj1SmmAU7C0A@mail.gmail.com> Hey, On Sat, Jul 16, 2011 at 6:40 PM, Benji York <benji at benjiyork.com> wrote: > On Sat, Jul 16, 2011 at 6:58 AM, Martijn Faassen <faassen at startifact.com> wrote: >> I wonder whether there are tooling solutions possible to detect this before >> it's too late. A public log of what got removed would be useful so people >> can keep an eye on things - but for this to be caught it would mean that the >> log would need to include recreations as well. > > Being a buildout user, if I were to tackle that I'd add something along > the lines of SSH's warnings when a host fingerprint changes. ?I.e., > require that package hashes be given (much like you can require that > versions be specified) and check those on download. Yes, for changes this would be possible (assuming hashes). Removals by themselves are another problem, though. Regards, Martijn From tjreedy at udel.edu Mon Jul 18 22:43:34 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 18 Jul 2011 16:43:34 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivrqs7$ppb$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> Message-ID: <j025tl$d9g$1@dough.gmane.org> On 7/16/2011 6:58 AM, Martijn Faassen wrote: > Okay, so this scenario is possible: > > * developer of a popular package gets fed up for unknown reasons > > * removes his package from PyPI (not realizing the thing below) > > * someone else notices this and recreates the package maliciously pypi could prohibit the reuse of deleted package names. If a name was 'retired' for legal reasons, then it should stay retired anyway. -- Terry Jan Reedy From tjreedy at udel.edu Mon Jul 18 22:47:58 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 18 Jul 2011 16:47:58 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivrql8$oq2$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> <87aacfb0r4.fsf@benfinney.id.au> <ivrql8$oq2$1@dough.gmane.org> Message-ID: <j0265t$fmh$1@dough.gmane.org> On 7/16/2011 6:54 AM, Martijn Faassen wrote: > So I have use cases: I can release code that relies on releases that can > disappear or can be replaced. I think this is bad for repeatability and > security. I'd like to see some improvements made. How would we make > these improvements? I've so far proposed three ideas: > > * PyPI not throwing away things after a grace period. Almost universally > rejected idea > > * an additional service, a mirror, that offers some repeatability > guarantees. Removal would need to go through channels, implying some > kind of custodianship I think people here are wary about. > > * better communication channels: a list of what's been removed, a list > of what's been deprecated. I can then write tools that help me maintain > my projects. It's not the same as the above ideas: old projects can > still break at the whim of people whose code I depend on, but it'll at > least help manage this issue. If packages have dependency list, then pypi could keep a ref count for each package and either not allow deletion of packages with a + count or require admin approval or at least inform the developer 'You are about to delete a package that other programs depend on: you sure you want to delete it?' followed 'Are you really really sure? Can we keep the file for a few months and give users a warning?" -- Terry Jan Reedy From mal at egenix.com Tue Jul 19 00:04:09 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Jul 2011 00:04:09 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <j025tl$d9g$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> Message-ID: <4E24ADD9.4080106@egenix.com> Terry Reedy wrote: > On 7/16/2011 6:58 AM, Martijn Faassen wrote: > >> Okay, so this scenario is possible: >> >> * developer of a popular package gets fed up for unknown reasons >> >> * removes his package from PyPI (not realizing the thing below) >> >> * someone else notices this and recreates the package maliciously > > pypi could prohibit the reuse of deleted package names. > If a name was 'retired' for legal reasons, then it should stay retired > anyway. Recycling of package names can very well have a real and honest background, e.g. if someone decides to give a package name to someone else for whatever reason. Happens in DNS all the time. BTW: To address your repeatability/security concerns, the tools you are using would also have to store the hash check sum of the downloaded packages together with the version. AFAIK, buildout only pins down versions, not MD5/SHA1 sums. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From faassen at startifact.com Tue Jul 19 18:11:23 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 19 Jul 2011 18:11:23 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <j0265t$fmh$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> <87aacfb0r4.fsf@benfinney.id.au> <ivrql8$oq2$1@dough.gmane.org> <j0265t$fmh$1@dough.gmane.org> Message-ID: <j04a9j$qr7$1@dough.gmane.org> On 07/18/2011 10:47 PM, Terry Reedy wrote: > If packages have dependency list, then pypi could keep a ref count for > each package and either not allow deletion of packages with a + count or > require admin approval or at least inform the developer 'You are about > to delete a package that other programs depend on: you sure you want to > delete it?' followed 'Are you really really sure? Can we keep the file > for a few months and give users a warning?" An interesting idea. Wouldn't work for code that isn't on PyPI (an internal project or whatever). It also would work less well with removing older versions: show it forbid throwing away 3.2 if there's a 3.3, or not? Requirements along the line of foo <= 3.2 exist, but are not very common, and cannot anticipate changes in future releases. (giving users a warning is another feature we don't have now in PyPI) Regards, Martijn From chris at simplistix.co.uk Tue Jul 19 19:46:38 2011 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 19 Jul 2011 18:46:38 +0100 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E24ADD9.4080106@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> Message-ID: <4E25C2FE.7050405@simplistix.co.uk> On 18/07/2011 23:04, M.-A. Lemburg wrote: > BTW: To address your repeatability/security concerns, the tools you are > using would also have to store the hash check sum of the downloaded > packages together with the version. AFAIK, buildout only pins down > versions, not MD5/SHA1 sums. I'm pretty sure there's a hashing extension for buildout downloads. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From mal at egenix.com Wed Jul 20 10:54:00 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Jul 2011 10:54:00 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E25C2FE.7050405@simplistix.co.uk> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> <4E25C2FE.7050405@simplistix.co.uk> Message-ID: <4E2697A8.1090505@egenix.com> Chris Withers wrote: > On 18/07/2011 23:04, M.-A. Lemburg wrote: >> BTW: To address your repeatability/security concerns, the tools you are >> using would also have to store the hash check sum of the downloaded >> packages together with the version. AFAIK, buildout only pins down >> versions, not MD5/SHA1 sums. > > I'm pretty sure there's a hashing extension for buildout downloads. You mean: an extension that allow pinning versions and hashes or just one that checks the downloads against the hashes provided by the index server (if it does) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 20 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From chris at simplistix.co.uk Wed Jul 20 10:55:09 2011 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 20 Jul 2011 09:55:09 +0100 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E2697A8.1090505@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> <4E25C2FE.7050405@simplistix.co.uk> <4E2697A8.1090505@egenix.com> Message-ID: <4E2697ED.8010901@simplistix.co.uk> On 20/07/2011 09:54, M.-A. Lemburg wrote: > You mean: an extension that allow pinning versions buildout allows version pinning out of the box. > just one that checks the downloads against the hashes provided by > the index server (if it does) ? yes, I think, but I'm not 100% cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From pje at telecommunity.com Wed Jul 20 16:09:44 2011 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 20 Jul 2011 10:09:44 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E2697ED.8010901@simplistix.co.uk> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> <4E25C2FE.7050405@simplistix.co.uk> <4E2697A8.1090505@egenix.com> <4E2697ED.8010901@simplistix.co.uk> Message-ID: <20110720141023.070F03A409B@sparrow.telecommunity.com> At 09:55 AM 7/20/2011 +0100, Chris Withers wrote: >On 20/07/2011 09:54, M.-A. Lemburg wrote: >>You mean: an extension that allow pinning versions > >buildout allows version pinning out of the box. > >>just one that checks the downloads against the hashes provided by >>the index server (if it does) ? > >yes, I think, but I'm not 100% Buildout uses setuptools, and setuptools checks the hashes provided by the index server. So, by default, buildout is probably checking them. From mal at egenix.com Wed Jul 20 16:29:23 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Jul 2011 16:29:23 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110720141023.070F03A409B@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> <4E25C2FE.7050405@simplistix.co.uk> <4E2697A8.1090505@egenix.com> <4E2697ED.8010901@simplistix.co.uk> <20110720141023.070F03A409B@sparrow.telecommunity.com> Message-ID: <4E26E643.1060902@egenix.com> > At 09:55 AM 7/20/2011 +0100, Chris Withers wrote: >> On 20/07/2011 09:54, M.-A. Lemburg wrote: >>> You mean: an extension that allow pinning versions >> >> buildout allows version pinning out of the box. Right, but does it also allow pinning down the hashes to make sure that the downloads are indeed the ones you expect ? That would be essential to provide security for buildout-based configurations that pull their data from servers not under control of the buildout user. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 20 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

From monitor at jacobian.org Fri Jul 1 02:05:27 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Thu, 30 Jun 2011 19:05:27 -0500 Subject: [Catalog-sig] [monit] pypi.python.org - Connection failed Message-ID: <1309478732.1@jacobian.org> Connection failed Service pypi.python.org Date: Thu, 30 Jun 2011 19:05:27 -0500 Action: alert Host: jacobian.org Description: failed protocol test [HTTP] at INET[pypi.python.org:80] via TCP Your faithful employee, monit From techtonik at gmail.com Fri Jul 1 08:30:31 2011 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 1 Jul 2011 09:30:31 +0300 Subject: [Catalog-sig] Tab Title customization and Tal to Django/Jinja transition In-Reply-To: References:

Message-ID: On Fri, Jul 1, 2011 at 3:01 AM, Richard Jones wrote: > On 30 June 2011 04:57, anatoly techtonik wrote: >> Index: pypi/templates/standard_template.pt >> =================================================================== >> - ? ? ? >> + ? ? ?<title tal:content="string:${data/title} : Python Package Index" /> > > Seems like a good idea. I've double-checked that the easy_install > "API" doesn't depend on this HTML. Done. Have you seen the second patch? It should be more cool. I've reattached it. > Can we stop the rather pointless arguing over templating engines now? Sure. Pointless arguing is a waste of time. I hoped to get an answer about superior features of TAL, which didn't seem as popular as Django/Jinja for me. Now I see that majority of PyPI devs are more familiar with TAL. So far my biggest concern is template readability. It will help if my editor had syntax highlighting for this XML language. All I need for that is XSD schema. Unfortunately, references contain only EBNF notation. -- anatoly t. -------------- next part -------------- Index: pypi/templates/display.pt =================================================================== --- pypi/templates/display.pt (revision 925) +++ pypi/templates/display.pt (working copy) @@ -4,6 +4,8 @@ xmlns:metal="http://xml.zope.org/namespaces/metal" metal:use-macro="standard_template/macros/page"> +<metal:fill fill-slot="title" tal:content="string:${data/name} ${data/version}: Python Package Index" /> + <metal:fill fill-slot="head"> <meta tal:condition="data/release/keywords | nothing" name="keywords" Index: pypi/templates/standard_template.pt =================================================================== --- pypi/templates/standard_template.pt (revision 925) +++ pypi/templates/standard_template.pt (working copy) @@ -8,7 +8,7 @@ <META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW" tal:condition="data/norobots"/> <meta content="text/html; charset=utf-8" http-equiv="content-type" /> <base tal:attributes="href data/FULL_PATH_INFO"/> - <title tal:content="string:Python Package Index : ${data/title}" /> + <title metal:define-slot="title" tal:content="Python Package Index : string:${data/title}" /> <meta tal:attributes="content data/keywords" /> <meta tal:attributes="content data/description" /> <link rel="alternate" type="application/rss+xml" title="RSS: 30 latest updates" href="http://www.python.org/pypi?:action=rss"/> Index: pypi/webui.py =================================================================== --- pypi/webui.py (revision 925) +++ pypi/webui.py (working copy) @@ -1348,7 +1348,6 @@ name=name, version=version, release=release, description=release.get('summary') or name, keywords=release.get('keywords', ''), - title=name + " " +version, requires=values('requires'), provides=values('provides'), obsoletes=values('obsoletes'), From richard at python.org Fri Jul 1 10:35:27 2011 From: richard at python.org (Richard Jones) Date: Fri, 1 Jul 2011 18:35:27 +1000 Subject: [Catalog-sig] Tab Title customization and Tal to Django/Jinja transition In-Reply-To: <BANLkTikvghHWqXpnDQPKs-u8E4AH7XjEqw@mail.gmail.com> References: <BANLkTik5PURsjjufC+VdL2zgiCOKobA=Nw@mail.gmail.com> <BANLkTinptZ-SNywGichZfKJz_49fTgT8yw@mail.gmail.com> <BANLkTikvghHWqXpnDQPKs-u8E4AH7XjEqw@mail.gmail.com> Message-ID: <BANLkTikv1gjD6nZucW8hjqWWn+DSTuKnKg@mail.gmail.com> On 1 July 2011 16:30, anatoly techtonik <techtonik at gmail.com> wrote: > On Fri, Jul 1, 2011 at 3:01 AM, Richard Jones <richard at python.org> wrote: >> On 30 June 2011 04:57, anatoly techtonik <techtonik at gmail.com> wrote: >>> Index: pypi/templates/standard_template.pt >>> =================================================================== >>> - ? ? ?<title tal:content="string:Python Package Index : ${data/title}" /> >>> + ? ? ?<title tal:content="string:${data/title} : Python Package Index" /> >> >> Seems like a good idea. I've double-checked that the easy_install >> "API" doesn't depend on this HTML. Done. > > Have you seen the second patch? It should be more cool. I've reattached it. I don't believe it's necessary, thanks. The site is consistent with the code as above. Richard From richard at python.org Fri Jul 1 02:01:10 2011 From: richard at python.org (Richard Jones) Date: Fri, 1 Jul 2011 10:01:10 +1000 Subject: [Catalog-sig] Tab Title customization and Tal to Django/Jinja transition In-Reply-To: <BANLkTik5PURsjjufC+VdL2zgiCOKobA=Nw@mail.gmail.com> References: <BANLkTik5PURsjjufC+VdL2zgiCOKobA=Nw@mail.gmail.com> Message-ID: <BANLkTinptZ-SNywGichZfKJz_49fTgT8yw@mail.gmail.com> On 30 June 2011 04:57, anatoly techtonik <techtonik at gmail.com> wrote: > Index: pypi/templates/standard_template.pt > =================================================================== > - ? ? ?<title tal:content="string:Python Package Index : ${data/title}" /> > + ? ? ?<title tal:content="string:${data/title} : Python Package Index" /> Seems like a good idea. I've double-checked that the easy_install "API" doesn't depend on this HTML. Done. Can we stop the rather pointless arguing over templating engines now? Richard From merwok at netwok.org Sat Jul 2 14:18:08 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 02 Jul 2011 14:18:08 +0200 Subject: [Catalog-sig] Tab Title customization and Tal to Django/Jinja transition In-Reply-To: <BANLkTinptZ-SNywGichZfKJz_49fTgT8yw@mail.gmail.com> References: <BANLkTik5PURsjjufC+VdL2zgiCOKobA=Nw@mail.gmail.com> <BANLkTinptZ-SNywGichZfKJz_49fTgT8yw@mail.gmail.com> Message-ID: <4E0F0C80.3060007@netwok.org> Le 01/07/2011 02:01, Richard Jones a ?crit : > Seems like a good idea. I've double-checked that the easy_install > "API" doesn't depend on this HTML. Done. Thanks Richard! still-finding-that-the-colon-is-strange?ly yours, ?ric From faassen at startifact.com Mon Jul 4 17:41:17 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 04 Jul 2011 17:41:17 +0200 Subject: [Catalog-sig] disallowing the removal of packages? Message-ID: <iusmso$6nl$1@dough.gmane.org> Hi there, My development project relied on a certain version of a package that is hosted on pypi. Unfortunately the package mainatainer had decided that certain older versions of the package were not longer supported and had removed them from PyPI altogether. This broke the build procedure for my project: I had carefully pinned down that I wanted this version (as that's what we tested with), and it wasn't available anymore. So, the removal of old packages from PyPI means that other people who rely on these packages with their installation instructions ("install Foo 3.3") or build tools (installation instructions, automated), suddenly have to deal with breakage. I thought perhaps PyPI can help and handle this problem at the source. What are the possible use cases for removing a release from PyPI? a) mistakes: the upload was accidental - we weren't supposed to release this at all! I.e. I uploaded private code that I never meant to distribute in the first place. b) actively hostile: the upload actually contains deliberately malicious code and protecting people against this *should* break their builds c) legal issues: some party claims that this code is not PyPI's to distribute in the first place, and for legal issues it'd need to be taken down. d) broken: security or other bugs that makes us want to loudly say, this is so broken, don't use it anymore e) cleaning up old stale unsupported stuff. I'd argue d) and e) are not up to the package maintainer to decide but to the person who integrates this package into their system. The person who integrates the package is the one who will need to make the judgment call to continue to use old unsupported or broken stuff. Integrators should be allowed to make such decisions in their own time at their own convenience; the package developer shouldn't be able to force such decisions by removing an old release. (We can think about better warning mechanisms: perhaps there could be some metadata or rule to decide when an old release is "deprecated" and then integration and installation tools could warn about this, but all this would be to help the integrator a better decision on what to do.) Use cases a), b) and c) are left. I think b) and c) should be handled by the PyPI administrators: this takes a judgment call and the package maintainer might not be involved in this at all. Use case a) is the tricky one. I could upload something by mistake, and then immediately discover it and decide to throw it out again. But I'd be fine if I weren't allowed removing a release anymore after a period of a month. It could be that I only discover my mistake after a month, when people have already started relying on this version, but in that case as a PyPI user I'd want the administrators to issue a judgement call there too. So, my proposal would be to allow people to remove a recent release freely, but after a period of (I don't know? A day? A week? A month?) removing the old release is not allowed anymore. What do people think? Regards, Martijn From jim at zope.com Mon Jul 4 19:04:49 2011 From: jim at zope.com (Jim Fulton) Date: Mon, 4 Jul 2011 13:04:49 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iusmso$6nl$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> Message-ID: <CAPDm-Fj9UTTyB_pgcxpadocmhCRneKKtgh9yKFqB1B=xMBEo2g@mail.gmail.com> On Mon, Jul 4, 2011 at 11:41 AM, Martijn Faassen <faassen at startifact.com> wrote: > Hi there, > > My development project relied on a certain version of a package that is > hosted on pypi. Unfortunately the package mainatainer had decided that > certain older versions of the package were not longer supported and had > removed them from PyPI altogether. > > This broke the build procedure for my project: I had carefully pinned down > that I wanted this version (as that's what we tested with), and it wasn't > available anymore. > > So, the removal of old packages from PyPI means that other people who rely > on these packages with their installation instructions ("install Foo 3.3") > or build tools (installation instructions, automated), suddenly have to deal > with breakage. > > I thought perhaps PyPI can help and handle this problem at the source. > > What are the possible use cases for removing a release from PyPI? > > a) mistakes: the upload was accidental - we weren't supposed to release this > at all! I.e. I uploaded private code that I never meant to distribute in the > first place. > > b) actively hostile: the upload actually contains deliberately malicious > code and protecting people against this *should* break their builds > > c) legal issues: some party claims that this code is not PyPI's to > distribute in the first place, and for legal issues it'd need to be taken > down. > > d) broken: security or other bugs that makes us want to loudly say, this is > so broken, don't use it anymore > > e) cleaning up old stale unsupported stuff. > > I'd argue d) and e) are not up to the package maintainer to decide but to > the person who integrates this package into their system. The person who > integrates the package is the one who will need to make the judgment call to > continue to use old unsupported or broken stuff. Integrators should be > allowed to make such decisions in their own time at their own convenience; > the package developer shouldn't be able to force such decisions by removing > an old release. > > (We can think about better warning mechanisms: perhaps there could be some > metadata or rule to decide when an old release is "deprecated" and then > integration and installation tools could warn about this, but all this would > be to help the integrator a better decision on what to do.) > > Use cases a), b) and c) are left. I think b) and c) should be handled by the > PyPI administrators: this takes a judgment call and the package maintainer > might not be involved in this at all. > > Use case a) is the tricky one. I could upload something by mistake, and then > immediately discover it and decide to throw it out again. But I'd be fine if > I weren't allowed removing a release anymore after a period of a month. It > could be that I only discover my mistake after a month, when people have > already started relying on this version, but in that case as a PyPI user I'd > want the administrators to issue a judgement call there too. > > So, my proposal would be to allow people to remove a recent release freely, > but after a period of (I don't know? A day? A week? A month?) removing the > old release is not allowed anymore. > > What do people think? +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From pje at telecommunity.com Mon Jul 4 20:59:21 2011 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 04 Jul 2011 14:59:21 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iusmso$6nl$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> Message-ID: <20110704185935.71A623A404D@sparrow.telecommunity.com> At 05:41 PM 7/4/2011 +0200, Martijn Faassen wrote: >I'd argue d) and e) are not up to the package maintainer to decide >but to the person who integrates this package into their system. Sure... but that doesn't mean the package maintainer is obligated to support the integrator in that decision. >The person who integrates the package is the one who will need to >make the judgment call to continue to use old unsupported or broken >stuff. Integrators should be allowed to make such decisions in their >own time at their own convenience; the package developer shouldn't >be able to force such decisions by removing an old release. Then the package integrator should darn well keep their own copy instead of relying on it still being downloadable from a public server. Not keeping a file uploaded is not equal to forcing anybody else to do anything. Note that there is nothing in your proposal that keeps a package maintainer from simply never uploading packages to PyPI in the first place... and is likely to have the perverse effect of encouraging package authors who are concerned about this issue to make other hosting arrangements. (Certainly, if it looks like your proposal will be adopted, I would be strongly motivated to *immediately* remove any package from PyPI that I thought I might need to remove later, but would be unable to if the proposal were implemented!) In short, this proposal is asking PyPI to do a job that is properly done by either the web archive or your private backups. -1. From g.brandl at gmx.net Mon Jul 4 21:25:36 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 04 Jul 2011 21:25:36 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <20110704185935.71A623A404D@sparrow.telecommunity.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> Message-ID: <iut42g$l2b$1@dough.gmane.org> Am 04.07.2011 20:59, schrieb P.J. Eby: > At 05:41 PM 7/4/2011 +0200, Martijn Faassen wrote: >>I'd argue d) and e) are not up to the package maintainer to decide >>but to the person who integrates this package into their system. > > Sure... but that doesn't mean the package maintainer is obligated to > support the integrator in that decision. > > >>The person who integrates the package is the one who will need to >>make the judgment call to continue to use old unsupported or broken >>stuff. Integrators should be allowed to make such decisions in their >>own time at their own convenience; the package developer shouldn't >>be able to force such decisions by removing an old release. > > Then the package integrator should darn well keep their own copy > instead of relying on it still being downloadable from a public > server. Not keeping a file uploaded is not equal to forcing anybody > else to do anything. > > Note that there is nothing in your proposal that keeps a package > maintainer from simply never uploading packages to PyPI in the first > place... and is likely to have the perverse effect of encouraging > package authors who are concerned about this issue to make other > hosting arrangements. > > (Certainly, if it looks like your proposal will be adopted, I would > be strongly motivated to *immediately* remove any package from PyPI > that I thought I might need to remove later, but would be unable to > if the proposal were implemented!) > > In short, this proposal is asking PyPI to do a job that is properly > done by either the web archive or your private backups. -1. I have to agree. Any hosting service that doesn't allow me to remove uploaded content looks extremely fishy to me. I think this list has already discussed the issue of people using PyPI as a dependency for deployment in production, and recommended the use of private package mirrors. Georg From jacob at jacobian.org Mon Jul 4 21:34:45 2011 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Mon, 4 Jul 2011 14:34:45 -0500 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iut42g$l2b$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> Message-ID: <CAK8PqJE2W9bfDXROQE0XPmfW0s=u7karC-UdqCg7rizgPJfvdA@mail.gmail.com> On Mon, Jul 4, 2011 at 2:25 PM, Georg Brandl <g.brandl at gmx.net> wrote: > I have to agree. ?Any hosting service that doesn't allow me to remove > uploaded content looks extremely fishy to me. It pains me, but I have to agree as well. If I had a nickel for each time I've been bitten by a package disappearing off PyPI... well, I'd be able to buy a few beers, at least. I believe, as I've argued in the past, that PyPI's raison d'?tre is to be a cataloging resource *for package authors* ? if we want to prevent fragmentation, then we have to make PyPI the "path of least resistance" for packagers. This means PyPI can't be in the business of making these sorts of decisions on authors' behalf. Not that we shouldn't get upset when a package vanishes. I'd be all for some public chastisement in those cases... (80% joking). Jacob From fuzzyman at gmail.com Mon Jul 4 21:48:38 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Mon, 4 Jul 2011 20:48:38 +0100 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iut42g$l2b$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> Message-ID: <CAKCKLWzKac6YBeHSDuASgyi2sFjux+GLmgOyAt+PY2pthqx28A@mail.gmail.com> On 4 July 2011 20:25, Georg Brandl <g.brandl at gmx.net> wrote: > Am 04.07.2011 20:59, schrieb P.J. Eby: > > At 05:41 PM 7/4/2011 +0200, Martijn Faassen wrote: > >>I'd argue d) and e) are not up to the package maintainer to decide > >>but to the person who integrates this package into their system. > > > > Sure... but that doesn't mean the package maintainer is obligated to > > support the integrator in that decision. > > > > > >>The person who integrates the package is the one who will need to > >>make the judgment call to continue to use old unsupported or broken > >>stuff. Integrators should be allowed to make such decisions in their > >>own time at their own convenience; the package developer shouldn't > >>be able to force such decisions by removing an old release. > > > > Then the package integrator should darn well keep their own copy > > instead of relying on it still being downloadable from a public > > server. Not keeping a file uploaded is not equal to forcing anybody > > else to do anything. > > > > Note that there is nothing in your proposal that keeps a package > > maintainer from simply never uploading packages to PyPI in the first > > place... and is likely to have the perverse effect of encouraging > > package authors who are concerned about this issue to make other > > hosting arrangements. > > > > (Certainly, if it looks like your proposal will be adopted, I would > > be strongly motivated to *immediately* remove any package from PyPI > > that I thought I might need to remove later, but would be unable to > > if the proposal were implemented!) > > > > In short, this proposal is asking PyPI to do a job that is properly > > done by either the web archive or your private backups. -1. > > I have to agree. Any hosting service that doesn't allow me to remove > uploaded content looks extremely fishy to me. > > Sourceforge for a long time had a policy against removing any released source packages (and may still have that policy as far as I know). All the best, Michael > I think this list has already discussed the issue of people using PyPI > as a dependency for deployment in production, and recommended the use > of private package mirrors. > > Georg > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110704/7d6438f9/attachment-0001.html> From faassen at startifact.com Mon Jul 4 21:56:15 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 4 Jul 2011 21:56:15 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <20110704185935.71A623A404D@sparrow.telecommunity.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> Message-ID: <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> Hi there, If I hand a project that relies on 100 dependencies to someone, I prefer not to have to hand private copies of all those 100 dependencies to that someone as well just in case a package maintainer removes a version from PyPI. It just makes my life harder. I'd prefer to rely on a centrally maintained infrastructure in that case. I could, in theory, make my own backup arrangement where I basically replicate PyPI to keep all previous releases online forever, and then rely on that. If you place your packages on PyPI, your packages would end up on this backup arrangement of mine. But it seems odd not to do the right thing at the source. So came to discuss it there. So anyway, I'm discussing use cases. Let's get back to that. I think you have a very different view of what PyPI is for, or could be for, than I do. Is PyPI a service for Python developers to find reusable code? Is PyPI a hosting site for Python developers to publish their code online? Does PyPI support integrators? Or is it more like a hosting site where people can do whatever they want? How much is this like, say, Debian and how much is this like a developer's website? > (Certainly, if it looks like your proposal will be adopted, I would be strongly motivated to *immediately* remove any package from PyPI > that I thought I might need to remove later, but would be unable to if the proposal were implemented!) [why taking such an aggressive stance?] Clearly you have a concept of which packages or releases you might need to remove later: could you state your motivations behind removing packages or releases? Perhaps there's a use case I missed in the above, or perhaps, again, you have a different philosophy of what PyPI is for. Regards, Martijn From faassen at startifact.com Mon Jul 4 22:02:58 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 04 Jul 2011 22:02:58 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iut42g$l2b$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> Message-ID: <iut67b$1c0$1@dough.gmane.org> Hi there, On 07/04/2011 09:25 PM, Georg Brandl wrote: > I have to agree. Any hosting service that doesn't allow me to remove > uploaded content looks extremely fishy to me. That's the question: is PyPI like a web hosting service? Or is PyPI something else, or something else as well? Is wikipedia a hosting service, or something else? Should it be keeping old revisions of documents? Is Debian a hosting service, or something else? Should it be keeping old revisions of packages? > I think this list has already discussed the issue of people using PyPI > as a dependency for deployment in production, and recommended the use > of private package mirrors. Why should these package mirrors have to be private? Why can't we share one? Regards, Martijn From g.brandl at gmx.net Mon Jul 4 22:03:41 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 04 Jul 2011 22:03:41 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAKCKLWzKac6YBeHSDuASgyi2sFjux+GLmgOyAt+PY2pthqx28A@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> <CAKCKLWzKac6YBeHSDuASgyi2sFjux+GLmgOyAt+PY2pthqx28A@mail.gmail.com> Message-ID: <iut69o$17l$1@dough.gmane.org> Am 04.07.2011 21:48, schrieb Michael Foord: > > In short, this proposal is asking PyPI to do a job that is properly > > done by either the web archive or your private backups. -1. > > I have to agree. Any hosting service that doesn't allow me to remove > uploaded content looks extremely fishy to me. > > > > Sourceforge for a long time had a policy against removing any released source > packages (and may still have that policy as far as I know). I've just tried, I was able to remove all files from an old project (it is now hosted on PyPI only) just fine. Georg From faassen at startifact.com Mon Jul 4 22:06:45 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 04 Jul 2011 22:06:45 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAK8PqJE2W9bfDXROQE0XPmfW0s=u7karC-UdqCg7rizgPJfvdA@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <iut42g$l2b$1@dough.gmane.org> <CAK8PqJE2W9bfDXROQE0XPmfW0s=u7karC-UdqCg7rizgPJfvdA@mail.gmail.com> Message-ID: <iut6ee$3ce$1@dough.gmane.org> On 07/04/2011 09:34 PM, Jacob Kaplan-Moss wrote: > I believe, as I've argued in the past, that PyPI's raison > d'?tre is to be a cataloging resource *for package authors* ? if we > want to prevent fragmentation, then we have to make PyPI the "path of > least resistance" for packagers. This means PyPI can't be in the > business of making these sorts of decisions on authors' behalf. > > Not that we shouldn't get upset when a package vanishes. I'd be all > for some public chastisement in those cases... (80% joking). So can we then have *another* service running on python.org that keeps all older revisions of packages around, for the convience of *package users*? Because this is putting an extra burden on package users. And also on package authors, by the way: What if I release package foo, which needs package bar, version 2.1 or version 2.2. Now the author of package bar removes version 2.1 and 2.2. Now nobody can use package foo anymore. You can't argue that these people should've made a private mirror, as this applies to all *future* users of package foo either. This might be annoying to the author of foo... And the packager of foo, who *is* using a private mirror as you are all recommending, might not find out very quickly that this situation has emerged. :) Regards, Martijn From faassen at startifact.com Mon Jul 4 22:13:26 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 4 Jul 2011 22:13:26 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E121AAB.5010305@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> Message-ID: <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> Hi there, On Mon, Jul 4, 2011 at 9:55 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> What do people think? > > I agree with PJE, Jacob, and Georg: package owners need > to have absolute freedom to delete content at any time, > or even replace it with new content. Okay, if there's a consensus among PyPI maintainers that doing such things should be absolutely free to developers, what about a central backup server that does keep such old copies around that just clones PyPI? Evidently I'll have to find people who are interested in that. > I disagree about possible reactions, though: your first action > should be to ask the package author to bring the old version > back. Maybe they didn't know you were still using it. That was of course my first reaction, and that of several others in the same situation, a few weeks ago when this came up. One release was restored, but the release several of us were actually using wasn't. It was an easy upgrade, but I'd have preferred to prefer to deal with this situation at a time of my own choosing. So I figured I'd just prefer to use a system where such a situation was impossible, and it was clearly a problem others were having too, so can be solved centrally. So anyway, I'm dropping my proposal, as it's going nowhere. I'll submit another proposal in a few minutes. But I am genuinely curious about the use cases behind allowing package authors to have absolute freedom, by the way - there's something I am not understanding. Is this because it is thought that otherwise developers won't use PyPI at all? It's clear PJE is one such developer, but I'm trying to understand *why*. It can't be that it's considered good practice to change the contents of an older release, or to remove one, right? It seems positively dangerous to allow people to arbitrarily replace old packages with new content - installation instructions will be totally broken, and there are some security risks as well. And this freedom doesn't seem to offer much more control to developers, as once the packages are on PyPI, people can make copies elsewhere. So what is the motivation behind allowing this freedom? Purely the thought that developers otherwise won't want to use PyPI? But why do we want people to use PyPI in the first place if not to allow a convenient reuse of this code? I'm missing something here... Regards, Martijn From martin at v.loewis.de Mon Jul 4 21:55:23 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Jul 2011 21:55:23 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iusmso$6nl$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> Message-ID: <4E121AAB.5010305@v.loewis.de> > What do people think? I agree with PJE, Jacob, and Georg: package owners need to have absolute freedom to delete content at any time, or even replace it with new content. If you don't trust the author to keep old versions available, you have to make your own copy. Make sure you don't violate the usage agreement of the software when doing so. I disagree about possible reactions, though: your first action should be to ask the package author to bring the old version back. Maybe they didn't know you were still using it. Regards, Martin From faassen at startifact.com Mon Jul 4 22:31:31 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 04 Jul 2011 22:31:31 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI Message-ID: <iut7ss$bmc$1@dough.gmane.org> Hi there, Is there any interest in running an immutable mirror of PyPI on python.org as a service to package users? What it would do is mirror the PyPI index and packages, with one difference: releases and packages once mirrored will be mirrored indefinitely. It will not accept changes of existing releases, or removal of existing releases from the mirror. Instead, it would keep an archive of these forever. To deal with cases where people make an upload by mistake, there could be a "window of removal", however, where removal is accepted if a release is not older than a certain age. Is there perhaps already mirroring code that can be used to create such a service? The motivation is to share a service that many of us are using PyPI for already: a way to conveniently share packages without having to make local backups or distribute local copies to all people who use our project. To reliably share packages the current PyPI is not sufficient, as PyPI has a philosophy of being a hosting site for packagers and therefore should allow package maintainers to freely change or remove previous releases at any point in time. Such an immutable mirror would be useful to package developers as well: you can release package a that depends on package b. You can then know that package b can't just be removed or modified, so that people who download your package a from the mirror can be guaranteed to always have access to the same package b that you tested your code with yourself. There would need to be a mechanism for the mirror administrators to remove releases on rare occasions where this might be needed for reasons of security or legality. Regards, Martijn From lac at openend.se Mon Jul 4 22:39:12 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 04 Jul 2011 22:39:12 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: Message from Martijn Faassen <faassen@startifact.com> of "Mon, 04 Jul 2011 21:56:15 +0200." <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com><CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> Message-ID: <201107042039.p64KdCwP009787@theraft.openend.se> Idea: When somebody wants to remove a package, they get a message 'are you sure you want to do this? Other people may be relying on this package. Wouldn't it be kinder to them to give them a depreciation period?' And then, either kick in a depreciation period -- during which time if Martijn dl the package e gets told that it is deprecated, and is going away in XXX (reasonable time to be worked out later, for the purposes of this article, say 6 weeks). Then in 6 weeks the package automatically goes away. Or if the author says, "Hell yes, delete the thing now!" than that happens instead. Laura From faassen at startifact.com Mon Jul 4 22:49:19 2011 From: faassen at startifact.com (Martijn Faassen) Date: Mon, 4 Jul 2011 22:49:19 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <201107042039.p64KdCwP009787@theraft.openend.se> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <faassen@startifact.com> <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> <201107042039.p64KdCwP009787@theraft.openend.se> Message-ID: <CAGT4ZFgw0jvOmcgJ9PvhQhSb2kS5Z39=vShRBv-BJbrCqJFnMw@mail.gmail.com> Hi there, On Mon, Jul 4, 2011 at 10:39 PM, Laura Creighton <lac at openend.se> wrote: > When somebody wants to remove a package, they get a message > 'are you sure you want to do this? ?Other people may be relying > on this package. This would, I imagine, be an improvement. Anyway, it's not that this is happening a *lot*; it's in fact extremely rare. This is what makes it so tempting to use PyPI in the way I (and many others) are using it. Perhaps with a warning like this it would not happen at all anymore. But you can't rely on it. > Wouldn't it be kinder to them to give them > a depreciation period?' > ?And then, either kick in a depreciation > period -- during which time if Martijn dl the package e gets told > that it is deprecated, and is going away in XXX (reasonable time to > be worked out later, for the purposes of this article, say 6 weeks). > Then in 6 weeks the package automatically goes away. ?Or if the > author says, "Hell yes, delete the thing now!" than that happens > instead. A communication channel for package maintainers to tell package users "hey, this has a really serious security bug!" or "this is deprecated" would be useful. The package homepage on PyPI can be used for that, of course, though perhaps isn't perfect as people who are using your package indirectly might not ever see it. But even if we had a deprecation mechanism, it wouldn't exactly solve my use case, as it'd still mean that I would need to distribute my own backups of all packages I'm using to be able to reliably replicate an installation somewhere else. Sometimes this is not a burden, and often it's wise to do so anyway, but frequently (say with a bunch of geographically distributed developers) it is a lot more convenient to be able to rely on a central infrastructure. It's not like doing this for dependencies is unheard of either - people are relying on content distribution networks for Javascript dependencies, for instance. Anyway, not changing or removing old releases just seems like the Right Thing To Do in general in software development; I'd like the tools to support this. Regards, Martijn From mal at egenix.com Tue Jul 5 00:48:23 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 05 Jul 2011 00:48:23 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iut7ss$bmc$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> Message-ID: <4E124337.4020001@egenix.com> Martijn Faassen wrote: > Hi there, > > Is there any interest in running an immutable mirror of PyPI on > python.org as a service to package users? AFAIK, gocept is running such a mirror for the own purposes. You might want to partner up with them. I can put you in touch with Christian Theune/GoCept if you like. In general, I find the idea to use a potentially volatile service for running buildout or similar tools a hazardous approach to software configuration management, esp. in production environments. Why don't you just download the packages you have tested and ship them with your application, bypassing all the network and usability issues of a dynamic catalog server ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 05 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From pje at telecommunity.com Tue Jul 5 02:18:31 2011 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 04 Jul 2011 20:18:31 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <201107042039.p64KdCwP009787@theraft.openend.se> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> <201107042039.p64KdCwP009787@theraft.openend.se> Message-ID: <20110705001846.BF3FF3A404D@sparrow.telecommunity.com> At 10:39 PM 7/4/2011 +0200, Laura Creighton wrote: >Idea: > >When somebody wants to remove a package, they get a message >'are you sure you want to do this? Other people may be relying >on this package. Wouldn't it be kinder to them to give them >a depreciation period?' And then, either kick in a depreciation >period -- during which time if Martijn dl the package e gets told >that it is deprecated, and is going away in XXX (reasonable time to >be worked out later, for the purposes of this article, say 6 weeks). >Then in 6 weeks the package automatically goes away. Or if the >author says, "Hell yes, delete the thing now!" than that happens >instead. This would be nice if there were a way to communicate to automated installation tools about this deprecation, in such a way that it reached people who could do something about it... and somehow magically motivated them to act. In practice, such a message would be mostly ignored by users until it actually broke their builds, at which time they will actually pay attention to what it says on the PyPI page. (Or, more likely, they will complain to somebody upstream of them who will then look at the PyPI page.) To put it another way, if you are removing a package because you really don't want people using it unless they have an excellent and *considered* reason for doing so (e.g. a version with security issues), you're not going to get anybody to STOP using that version unless you break their mindless automated build. That's the only way you're going to get their (motivated) attention pointed at whatever needs changing. That doesn't mean deprecation isn't a good idea, though. Certainly, the build tools out there could be updated to flag warnings about deprecated packages, and people with projects big enough to have QA (i.e., the folks who most likely care about this issue anyway) probably WOULD pay attention to deprecation warnings. (I don't think I'd object to using a deprecation process as the normal way of removing a package, but on principle I believe an author should have the ability to remove their uploads for any reason or no reason.) From martin at v.loewis.de Tue Jul 5 09:38:54 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 09:38:54 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> Message-ID: <4E12BF8E.7060901@v.loewis.de> Am 04.07.2011 21:56, schrieb Martijn Faassen: > How much is this like, say, Debian > and how much is this like a developer's website? It's completely unlike Debian: - in Debian, you have package maintainers that perform the packaging, deal with bug reports (despite not being the original author), and decide which upstream release is put into what Debian release. Nobody takes that role in PyPI. - in Debian, you have the DFSG and other policy documents determining what licenses are allowed, and how software must behave. In PyPI, the only policy is that it must be a Python package. I dom't know what you mean by "developer's website", so I can't comment on whether PyPI is one. Regards, Martin From martin at v.loewis.de Tue Jul 5 09:51:53 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 09:51:53 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> Message-ID: <4E12C299.6000807@v.loewis.de> > But I am genuinely curious about the use cases behind allowing package > authors to have absolute freedom, by the way - there's something I am > not understanding. Is this because it is thought that otherwise > developers won't use PyPI at all? In general, PyPI prefers to not set policies when we can avoid it. There was a long discussion on what package names should be allowed (generally and also about specific package names), and the conclusion was always that package authors can put any content on PyPI as long as it's Python-related, and not spam or illegal. It's only consequential that there shouldn't be a policy on whether a package can be deleted. I *personally* think you are exaggerating your case. If you are using a package, avoid sticking to a specific version of the software, and instead write the package in a way that will work with many versions of the library. If the library is too unstable, don't use it at all. If the library has an incompatible change, report it as a bug; if the author can bring convincing arguments why there must be that change, cope with it. IOW, just don't use old versions when newer versions are available and maintained. > It can't be that it's considered good practice to change the contents > of an older release, or to remove one, right? Right. And it can't be considered good practice to rely on software that follows bad practice, right? Don't use software that deletes old releases while making unjustified incompatible changes in new releases, or otherwise doesn't provide clear backwards compatibility statements and migration strategies. > It seems positively > dangerous to allow people to arbitrarily replace old packages with new > content - installation instructions will be totally broken, and there > are some security risks as well. Again, you, as the package user, need to evaluate these risks. If you trust the author, you don't need to worry. If you don't trust the author, either don't use it, or take countermeasures (like inspecting all code before using it, and using hashsums in automated deployments). Regards, Martin From martin at v.loewis.de Tue Jul 5 09:55:15 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 09:55:15 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFgw0jvOmcgJ9PvhQhSb2kS5Z39=vShRBv-BJbrCqJFnMw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <20110704185935.71A623A404D@sparrow.telecommunity.com> <faassen@startifact.com> <CAGT4ZFg2xR3RqNf+vnBEzygZqi25i_5ZWTm-F5KbSS_bYh+=DQ@mail.gmail.com> <201107042039.p64KdCwP009787@theraft.openend.se> <CAGT4ZFgw0jvOmcgJ9PvhQhSb2kS5Z39=vShRBv-BJbrCqJFnMw@mail.gmail.com> Message-ID: <4E12C363.90403@v.loewis.de> > A communication channel for package maintainers to tell package users > "hey, this has a really serious security bug!" or "this is deprecated" > would be useful. The package homepage on PyPI can be used for that, of > course, though perhaps isn't perfect as people who are using your > package indirectly might not ever see it. This may be a case where actually replacing an old release might be useful: you could put an actual DeprecationWarning into the code, or at least print a message in setup.py. This would increase the chance that anybody who has hard-coded the library version might see it. Regards, Martin From m.van.rees at zestsoftware.nl Tue Jul 5 13:16:39 2011 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Tue, 05 Jul 2011 13:16:39 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E12C299.6000807@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> Message-ID: <iuurqo$hd1$1@dough.gmane.org> Op 05-07-11 09:51, "Martin v. L?wis" schreef: > I *personally* think you are exaggerating your case. If you are using > a package, avoid sticking to a specific version of the software, and > instead write the package in a way that will work with many versions > of the library. If the library is too unstable, don't use it at all. > If the library has an incompatible change, report it as a bug; if > the author can bring convincing arguments why there must be that > change, cope with it. > > IOW, just don't use old versions when newer versions are available > and maintained. In my case, usually when I start a new (client) project for which I need package foo I use the latest release 1.0. The problems is that most of the time I do not know yet if a new version (whether that is going to be 1.0.1, 1.1 or 2.0) is going to cause backwards compatibility problems. And this may be the first time I am using package foo so I do not know yet if this author prefers to actively delete old versions. FWIW, I am using a pypi mirror created with collective.eggproxy as an index so that packages I use in my buildouts are available on that mirror (so only actually used packages are mirrored). This shields me at least partially from these problems. Cheers, -- Maurits van Rees Web App Programmer at Zest Software: http://zestsoftware.nl Personal website: http://maurits.vanrees.org/ From faassen at startifact.com Tue Jul 5 13:48:12 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 13:48:12 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E12C299.6000807@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> Message-ID: <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 9:51 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > I *personally* think you are exaggerating your case. If you are using > a package, avoid sticking to a specific version of the software, and > instead write the package in a way that will work with many versions > of the library. If the library is too unstable, don't use it at all. > If the library has an incompatible change, report it as a bug; if > the author can bring convincing arguments why there must be that > change, cope with it. I have about 5 years experience doing this. We started out doing it the way you suggested it in 2007. It didn't work. We got constant breakage. Users of your collection of packages would have to be lucky to download them on a day when it was all working perfectly. Everybody developing these packages were intelligent people working in good faith trying to preserve backwards compatibility, but things broke anyway. If you maintain the collection of packages you are willing to try upgrades to new versions of libraries and run the test suits to see whether it works, but if you then hand this collection of packages to other people that might start use it at unpredictable times, you want to hand them something that is guaranteed as much as possible to work. This applies to development teams when you just want to start working instead of having to deal with "oh this upgrade broke my build" all the time, and it also applies to open source projects which hand out a large collection of packages to users. [repeated] > I *personally* think you are exaggerating your case. ... > IOW, just don't use old versions when newer versions are available > and maintained. This leads me to conclude that I personally think you have not the same kind of experience with this issue as I do... I think you haven't been using packages on a scale I have been doing for years now. >> It can't be that it's considered good practice to change the contents >> of an older release, or to remove one, right? > > Right. And it can't be considered good practice to rely on software > that follows bad practice, right? I can't predict this in advance. I have been using this software for years, and suddenly it was removed. It's also rather facile of you to say that I can easily switch to something else when this happens - sometimes it's the only game in town. > Don't use software that deletes old releases while making unjustified > incompatible changes in new releases, or otherwise doesn't provide > clear backwards compatibility statements and migration strategies. Sorry, I can't see the future like you do. I also use packages that have imperfect maintainers that sometimes make incompatible releases by mistake; compatibility issues can be very subtle. Anyway, you're just telling me very loudly I don't have a problem, right? You're convinced I am doing it the wrong way. So should I just go away? Regards, Martijn From faassen at startifact.com Tue Jul 5 13:59:13 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 13:59:13 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E124337.4020001@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> Message-ID: <iuuu85$vmr$1@dough.gmane.org> Hi there, On 07/05/2011 12:48 AM, M.-A. Lemburg wrote: > Martijn Faassen wrote: >> Is there any interest in running an immutable mirror of PyPI on >> python.org as a service to package users? > > AFAIK, gocept is running such a mirror for the own purposes. > You might want to partner up with them. > > I can put you in touch with Christian Theune/GoCept if you like. I know Christian quite well, I can easily contact them. I hadn't realized they were running an immutable mirror. > In general, I find the idea to use a potentially volatile service > for running buildout or similar tools a hazardous approach to > software configuration management, esp. in production environments. > > Why don't you just download the packages you have tested and > ship them with your application, bypassing all the network > and usability issues of a dynamic catalog server ? Yeah, that's one way to resolve this. It's just a lot more work to do this during development than updating a version number in a configuration file when you need a new version. Concerning relying on networked resources for the installation of tested packages, Linux distributions have been doing this for years; I don't think that's a fundamentally flawed approach. PyPI just is at this point where it works 99.99% percent of the time, but it allows sudden surprises to pop up. Regards, Martijn From mal at egenix.com Tue Jul 5 14:12:41 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 05 Jul 2011 14:12:41 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iuuu85$vmr$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> Message-ID: <4E12FFB9.40603@egenix.com> Martijn Faassen wrote: > Hi there, > > On 07/05/2011 12:48 AM, M.-A. Lemburg wrote: >> Martijn Faassen wrote: >>> Is there any interest in running an immutable mirror of PyPI on >>> python.org as a service to package users? >> >> AFAIK, gocept is running such a mirror for the own purposes. >> You might want to partner up with them. >> >> I can put you in touch with Christian Theune/GoCept if you like. > > I know Christian quite well, I can easily contact them. I hadn't > realized they were running an immutable mirror. Ok. >> In general, I find the idea to use a potentially volatile service >> for running buildout or similar tools a hazardous approach to >> software configuration management, esp. in production environments. >> >> Why don't you just download the packages you have tested and >> ship them with your application, bypassing all the network >> and usability issues of a dynamic catalog server ? > > Yeah, that's one way to resolve this. > > It's just a lot more work to do this during development than updating a > version number in a configuration file when you need a new version. Hmm, the testing involved with upgrading to a new release is usually a lot more work than importing the release into a repository ;-) FWIW: We've been using the repository approach for many years now and it has never failed on us. Even doing a 1GB SVN checkout doesn't really take long if you're doing this on a server which is directly connected to the Internet. And after the checkout you can be sure that the production is running the exact same version as the one you've tested - a guarantee that tools such as buildout, which download the packages and then build them on the target machine, cannot provide. YMMV, though. > Concerning relying on networked resources for the installation of tested > packages, Linux distributions have been doing this for years; I don't > think that's a fundamentally flawed approach. Sure, but those are distributions of collections of packages that are known to work together (most of the time), not catalogs like PyPI, which don't provide any compatibility or availability guarantees. I was essentially suggesting to setup your own distribution or distribution server to avoid the issues with missing compatibility checks or loss of availability of a resource. > PyPI just is at this point where it works 99.99% percent of the time, > but it allows sudden surprises to pop up. I think that's just a coincident, not really a feature :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 05 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From faassen at startifact.com Tue Jul 5 15:14:26 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 15:14:26 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E12FFB9.40603@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> Message-ID: <CAGT4ZFgtgUamVh9Re-khmqB1_RDBKFAZ8NA4FtgJ_sXYV21gMw@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 2:12 PM, M.-A. Lemburg <mal at egenix.com> wrote: [snip] > Sure, but those are distributions of collections of packages > that are known to work together (most of the time), not catalogs > like PyPI, which don't provide any compatibility or availability > guarantees. > > I was essentially suggesting to setup your own distribution or > distribution server to avoid the issues with missing compatibility > checks or loss of availability of a resource. Yes, it's something to consider, and it's not too difficult to convert a project to a bunch of downloaded tarballs before deployment, but it's less friendly for sharing projects between developers; I'd have to maintain a separate distribution of lots of packages per project in my own version control system, once per project, right? Or I have to start maintaining my own distribution server. I don't want to maintain my own distribution server if I can share it. An immutable pypi would be a good distribution server, and mirroring is already practiced for PyPI and that would help with availability. It'd be a shared way to solve this problem as opposed to be everybody solving the problem for themselves. >> PyPI just is at this point where it works 99.99% percent of the time, >> but it allows sudden surprises to pop up. > > I think that's just a coincident, not really a feature :-) Yes, I realize that, that's why I initiated this discussion in the first place. It's only a few small steps away from a shared solution. Regards, Martijn From jim at zope.com Tue Jul 5 15:52:37 2011 From: jim at zope.com (Jim Fulton) Date: Tue, 5 Jul 2011 09:52:37 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E12FFB9.40603@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> Message-ID: <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> On Tue, Jul 5, 2011 at 8:12 AM, M.-A. Lemburg <mal at egenix.com> wrote: ... >> PyPI just is at this point where it works 99.99% percent of the time, >> but it allows sudden surprises to pop up. > > I think that's just a coincident, not really a feature :-) Actually, it's not. Within the Zope community, we early on realized the value of PyPI and setuptools as a collaboration tool and also realized that to fully realize it's potential, we'd need a policy of not removing old releases from PyPI. This policy was promulgated within the Zope community, has been followed, and has proven valuable. Of course, we still get burned from time to time when old versions of other packages we use disappear. I really don't intend to get embroiled in this debate, I've already offered a +1 to Martijn's proposal, but I will additionally offer a some opinions: - IMO, PyPI should be more for package consumers than for producers. I really have trouble fathoming the idea that a package index/catalog of primarily for producers. Perhaps I'm being dense. - A lot of PyPI's value is as a collaboration tool. It's presense, and more so with setuptools and its spawn, has made dynamic application assembly possible and reuse far more practical. - While I endorsed Martijn's idea, I think a voluntary system would be just as good and probably more palatable. I don't think we need a depracation system. There's rarely a good reason to remove an old package. Old releases are hidden from the human interface and automated tools will pick new releases unless told otherwise. I think if people understood that people were relying on old releases, they wouldn't remove them. IMO, it would be enough to warn someone about to delete that someone might be depending on the release and that deleting it could cause them hassle. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From faassen at startifact.com Tue Jul 5 16:18:54 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 16:18:54 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> Message-ID: <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.gmail.com> Hi there, Thanks for your input, Jim. > - IMO, PyPI should be more for package consumers than for producers. I > really have trouble fathoming the idea that a package index/catalog > of primarily for producers. Perhaps I'm being dense. Yes, I am being dense too, but then our perspective would be similar given our shared experiences. It seems clear from the responses by others that PyPI is seen as primarily for producers, not consumers of packages. That's why I asked whether PyPI is primarily a hosting site for developers (as opposed to something like Debian or Wikipedia which have notions about a collaborative effort of some kind, and care about preserving history). >From what I can interpret: PyPI is there so developers don't have to put the packages on their own web page, and to easily advertise that their packages and releases exist. Any use by consumers of PyPI for reuse of packages is secondary and really they should implement their own systems for reusing packages that does not rely on PyPI beyond an initial download. Is this a correct interpretation? Some more elaboration on the thinking behind this principle would be welcome. On Tue, Jul 5, 2011 at 3:52 PM, Jim Fulton <jim at zope.com> wrote: > ?IMO, it would be enough to warn someone about to delete that someone > ?might be depending on the release and that deleting it could cause > ?them hassle. Yes, this would indeed be helpful, as I expressed elsewhere. Regards, Martijn From lists at zopyx.com Tue Jul 5 16:25:40 2011 From: lists at zopyx.com (Andreas Jung) Date: Tue, 05 Jul 2011 16:25:40 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <iusmso$6nl$1@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> Message-ID: <4E131EE4.2010700@zopyx.com> Martijn Faassen wrote: > Hi there, > > My development project relied on a certain version of a package that is > hosted on pypi. Unfortunately the package mainatainer had decided that > certain older versions of the package were not longer supported and had > removed them from PyPI altogether. > > This broke the build procedure for my project: I had carefully pinned > down that I wanted this version (as that's what we tested with), and it > wasn't available anymore. > > So, the removal of old packages from PyPI means that other people who > rely on these packages with their installation instructions ("install > Foo 3.3") or build tools (installation instructions, automated), > suddenly have to deal with breakage. > > I thought perhaps PyPI can help and handle this problem at the source. > > What are the possible use cases for removing a release from PyPI? > > a) mistakes: the upload was accidental - we weren't supposed to release > this at all! I.e. I uploaded private code that I never meant to > distribute in the first place. > > b) actively hostile: the upload actually contains deliberately malicious > code and protecting people against this *should* break their builds > > c) legal issues: some party claims that this code is not PyPI's to > distribute in the first place, and for legal issues it'd need to be > taken down. > > d) broken: security or other bugs that makes us want to loudly say, this > is so broken, don't use it anymore > > e) cleaning up old stale unsupported stuff. > > I'd argue d) and e) are not up to the package maintainer to decide but > to the person who integrates this package into their system. The person > who integrates the package is the one who will need to make the judgment > call to continue to use old unsupported or broken stuff. Integrators > should be allowed to make such decisions in their own time at their own > convenience; the package developer shouldn't be able to force such > decisions by removing an old release. > > (We can think about better warning mechanisms: perhaps there could be > some metadata or rule to decide when an old release is "deprecated" and > then integration and installation tools could warn about this, but all > this would be to help the integrator a better decision on what to do.) > > Use cases a), b) and c) are left. I think b) and c) should be handled by > the PyPI administrators: this takes a judgment call and the package > maintainer might not be involved in this at all. > > Use case a) is the tricky one. I could upload something by mistake, and > then immediately discover it and decide to throw it out again. But I'd > be fine if I weren't allowed removing a release anymore after a period > of a month. It could be that I only discover my mistake after a month, > when people have already started relying on this version, but in that > case as a PyPI user I'd want the administrators to issue a judgement > call there too. > > So, my proposal would be to allow people to remove a recent release > freely, but after a period of (I don't know? A day? A week? A month?) > removing the old release is not allowed anymore. First I have to second Martijn in all aspects (bascially because are coming from the same projects and facing similar problems). The main (meta) question is: is PyPI a the wild-west of the Python world where everyone can do with packages or do we want PyPI being a well-managed place for packages with a certain number of rules. Looking at the RSS feed each day makes me angry sometimes because PyPI feels like the dumpster of the Python world. The points Martijn raised are only one aspect of this question. -aj -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/a847a459/attachment.vcf> From lists at zopyx.com Tue Jul 5 16:35:51 2011 From: lists at zopyx.com (Andreas Jung) Date: Tue, 05 Jul 2011 16:35:51 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iut7ss$bmc$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> Message-ID: <4E132147.2040104@zopyx.com> Martijn Faassen wrote: > Hi there, > > Is there any interest in running an immutable mirror of PyPI on > python.org as a service to package users? Aren't the PyPI mirrors (based on top of the pep381 client) immutable? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/d6359920/attachment.vcf> From pje at telecommunity.com Tue Jul 5 16:42:56 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 05 Jul 2011 10:42:56 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.g mail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> Message-ID: <20110705144311.B7CD43A404D@sparrow.telecommunity.com> At 09:52 AM 7/5/2011 -0400, Jim Fulton wrote: >- IMO, PyPI should be more for package consumers than for producers. I > really have trouble fathoming the idea that a package index/catalog > of primarily for producers. Perhaps I'm being dense. The cardinal rule of successful groupware is that those who put information *into* the system must receive a proportional benefit for their efforts, or those who want to get information *out* of the system will not see anything coming out. From pje at telecommunity.com Tue Jul 5 16:51:03 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 05 Jul 2011 10:51:03 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.g mail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.gmail.com> Message-ID: <20110705145108.149DB3A404D@sparrow.telecommunity.com> At 04:18 PM 7/5/2011 +0200, Martijn Faassen wrote: >That's why I asked >whether PyPI is primarily a hosting site for developers (as opposed to >something like Debian or Wikipedia which have notions about a >collaborative effort of some kind, and care about preserving history). It's right there in the name: Python Package *Index*. In other words, it's an index of packages. Hosting the packages is optional. Heck, a package actually having any *code* to download at all is optional, since you can list a package you merely intend to develop. Or you might just have a revision control link, etc. So no, it's not a curated repository of code. If that were the case, it'd be better named the Python Code Repository or some such. The perspective that you have is influenced by your use case for PyPI; by nature, these other sorts of packages (i.e. ones without uploaded, released code) are ones you don't care about. But that doesn't mean they don't exist. In any case, you can't turn PyPI into PyCR by the simple expedient of disallowing deletions; you'd need actual curators and a fresh website that doesn't contain all the unreleased, unmaintained, un-existent, or un-open-source packages found on PyPI. From faassen at startifact.com Tue Jul 5 17:34:38 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 17:34:38 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110705145108.149DB3A404D@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.gmail.com> <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.g mail.com> <20110705145108.149DB3A404D@sparrow.telecommunity.com> Message-ID: <iuvas1$i38$1@dough.gmane.org> On 07/05/2011 04:51 PM, P.J. Eby wrote: > At 04:18 PM 7/5/2011 +0200, Martijn Faassen wrote: >> That's why I asked >> whether PyPI is primarily a hosting site for developers (as opposed to >> something like Debian or Wikipedia which have notions about a >> collaborative effort of some kind, and care about preserving history). > > It's right there in the name: Python Package *Index*. In other words, > it's an index of packages. Hosting the packages is optional. Heck, a > package actually having any *code* to download at all is optional, since > you can list a package you merely intend to develop. Or you might just > have a revision control link, etc. Something being an index doesn't necessarily imply that the index will behave in a certain way. For instance, if I have an index of a database table I rather expect that things mentioned in the index actually are kept in sync with things in the database table. It depends entirely on what kind of index it is trying to be. If PyPI's index followed some principles of immutability it could be used in different ways than if it isn't. > So no, it's not a curated repository of code. If that were the case, > it'd be better named the Python Code Repository or some such. Sure, if it turns out that is a more useful to people, it might be changed (given certain assumptions about what the word "index" applies) > The perspective that you have is influenced by your use case for PyPI; > by nature, these other sorts of packages (i.e. ones without uploaded, > released code) are ones you don't care about. But that doesn't mean they > don't exist. I'm not saying that my use case is the only use case for PyPI. I just submitted my use cases for discussion. This is why I was carefully analyzing use cases for package removal in the first place... I'm still missing a few, but people don't seem to want to share them. :) > In any case, you can't turn PyPI into PyCR by the simple expedient of > disallowing deletions; you'd need actual curators and a fresh website > that doesn't contain all the unreleased, unmaintained, un-existent, or > un-open-source packages found on PyPI. I think you misunderstand: I am the curator of my own list packages and versions. I just like using a shared central system by which those packages are shared (and many others I might want someday to add to my list). Using the existing PyPI repository for this works fairly well, but there are some issues, such as package changes and package removal. I thought that since others might have similar concerns, we could work together to offer some guarantees that would help external curators such as myself. It appears however that my use cases are not shared by most people responding in this thread. That doesn't mean that they're necessarily invalid use cases for PyPI, but I'm not holding out much hope after this reception. :) Regards, Martijn From faassen at startifact.com Tue Jul 5 17:38:47 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 17:38:47 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110705144311.B7CD43A404D@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.g mail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> Message-ID: <iuvb3p$js1$1@dough.gmane.org> On 07/05/2011 04:42 PM, P.J. Eby wrote: > At 09:52 AM 7/5/2011 -0400, Jim Fulton wrote: >> - IMO, PyPI should be more for package consumers than for producers. I >> really have trouble fathoming the idea that a package index/catalog >> of primarily for producers. Perhaps I'm being dense. > > The cardinal rule of successful groupware is that those who put > information *into* the system must receive a proportional benefit for > their efforts, or those who want to get information *out* of the system > will not see anything coming out. What's the proportional benefit for those who put stuff in the system? How much effort is involved in putting things into the system? How would immutability hinder the benefit? I don't understand. Regards, Martijn From faassen at startifact.com Tue Jul 5 17:42:51 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 17:42:51 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E132147.2040104@zopyx.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> Message-ID: <iuvbbe$lc2$1@dough.gmane.org> On 07/05/2011 04:35 PM, Andreas Jung wrote: > Martijn Faassen wrote: >> Is there any interest in running an immutable mirror of PyPI on >> python.org as a service to package users? > > > Aren't the PyPI mirrors (based on top of the pep381 client) > immutable? You tell me. Are they? How interesting. :) I think you're right! The package that disappeared from the main PyPI a few weeks ago is still available on b.pypi.python.org, at least. Wow, thanks! I had assumed the mirrors would remove things, but it seems they don't. I think that leaves the overwriting issue: if I were to release a version of foo 1.3, and then later upload a new version of foo 1.3, would the mirror be updated with the new version or not? Regards, Martijn From lists at zopyx.com Tue Jul 5 17:59:40 2011 From: lists at zopyx.com (Andreas Jung) Date: Tue, 05 Jul 2011 17:59:40 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iuvbbe$lc2$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> <iuvbbe$lc2$1@dough.gmane.org> Message-ID: <4E1334EC.5090101@zopyx.com> Martijn Faassen wrote: > On 07/05/2011 04:35 PM, Andreas Jung wrote: >> Martijn Faassen wrote: >>> Is there any interest in running an immutable mirror of PyPI on >>> python.org as a service to package users? >> >> >> Aren't the PyPI mirrors (based on top of the pep381 client) >> immutable? > > You tell me. Are they? How interesting. :) > > I think you're right! The package that disappeared from the main PyPI a > few weeks ago is still available on b.pypi.python.org, at least. > > Wow, thanks! :-P > > I had assumed the mirrors would remove things, but it seems they don't. > I think that leaves the overwriting issue: if I were to release a > version of foo 1.3, and then later upload a new version of foo 1.3, > would the mirror be updated with the new version or not? Not sure - MvL should know - never checked the pep381 sources in detail. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/2ab50fbf/attachment.vcf> From faassen at startifact.com Tue Jul 5 18:14:17 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 18:14:17 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E1334EC.5090101@zopyx.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> <iuvbbe$lc2$1@dough.gmane.org> <4E1334EC.5090101@zopyx.com> Message-ID: <iuvd6c$vq0$1@dough.gmane.org> Hi there, Hm, actually this is wrong. In fact, b.pypi.python.org appears to be seriously out of date, and c.pypi.python.org does seem to be synched. Regards, Martijn From faassen at startifact.com Tue Jul 5 18:20:24 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 05 Jul 2011 18:20:24 +0200 Subject: [Catalog-sig] b mirror out of date? Message-ID: <iuvdhr$4ff$1@dough.gmane.org> Hi there, I just noticed that the 'b' mirror doesn't seem to be updated as much as the others. It does provide the 'last-modified' entry that other mirrors do, though it doesn't appear on the top-level index page like with the other mirrors (this is probably only an implementation detail). When I go there manually, I see: 20110303T12:42:55 Meaning it doesn't seem to have been updated in months. Is this intentional? Regards, Martijn From jim at zope.com Tue Jul 5 18:19:27 2011 From: jim at zope.com (Jim Fulton) Date: Tue, 5 Jul 2011 12:19:27 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110705145108.149DB3A404D@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAGT4ZFgW2MkqC7YrD=BR3tEP_hS63U3u_7ez4uAGfGFvd6uOFg@mail.gmail.com> <20110705145108.149DB3A404D@sparrow.telecommunity.com> Message-ID: <CAPDm-Fj5iXh2SKE_4Ybi+1Y26LzXVKytgwJS4_NvNw0G5ETrxA@mail.gmail.com> On Tue, Jul 5, 2011 at 10:51 AM, P.J. Eby <pje at telecommunity.com> wrote: > At 04:18 PM 7/5/2011 +0200, Martijn Faassen wrote: >> >> That's why I asked >> whether PyPI is primarily a hosting site for developers (as opposed to >> something like Debian or Wikipedia which have notions about a >> collaborative effort of some kind, and care about preserving history). > > It's right there in the name: Python Package *Index*. ??In other words, it's > an index of packages. ??Hosting the packages is optional. ??Heck, a package > actually having any *code* to download at all is optional, since you can > list a package you merely intend to develop. ??Or you might just have a > revision control link, etc. Even if you buy that PyPI is *just* an index, when someone removes a release, they are removing the *index* data for it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From lists at zopyx.com Tue Jul 5 18:25:03 2011 From: lists at zopyx.com (Andreas Jung) Date: Tue, 05 Jul 2011 18:25:03 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iuvd6c$vq0$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> <iuvbbe$lc2$1@dough.gmane.org> <4E1334EC.5090101@zopyx.com> <iuvd6c$vq0$1@dough.gmane.org> Message-ID: <4E133ADF.3030501@zopyx.com> Martijn Faassen wrote: > Hi there, > > Hm, actually this is wrong. In fact, b.pypi.python.org appears to be > seriously out of date, and c.pypi.python.org does seem to be synched. which is running on my machine :) The root of the server shows you the last synchronization date. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/7e5b7850/attachment.vcf> From pydanny at gmail.com Tue Jul 5 19:33:19 2011 From: pydanny at gmail.com (Daniel Greenfeld) Date: Tue, 5 Jul 2011 10:33:19 -0700 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E131EE4.2010700@zopyx.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> Message-ID: <CAOoSJ_roGq4Dy1qSU445s_hKwt5g9gRuCFNFscpUOv2n6Wf2QQ@mail.gmail.com> Alright, my two cents: 1. I would love an eternal PyPI mirror. The only things that should go away are things that violate the law or have grave security risks. 2. You can't expect anyone on the planet to build applications that are robust enough to handle when a package or a particular release suddenly vanishes. For example, if I'm building a Bluebream app and zca.core goes away, then me and every other Zope developer is screwed. Same goes for PyQT, Django, and the rest of the Python universe. 3. IMHO, the index is for package maintainers AND consumers. Maintainers have a place that provides hosting and the ability to track usage, and consumers have a place that provides searching and a go-to place for install tools. If that wasn't the original intent during the inception of PyPI, then maybe the intent ought to be changed to match this new intention. Daniel Greenfeld On Tue, Jul 5, 2011 at 7:25 AM, Andreas Jung <lists at zopyx.com> wrote: > > > Martijn Faassen wrote: >> >> Hi there, >> >> My development project relied on a certain version of a package that is >> hosted on pypi. Unfortunately the package mainatainer had decided that >> certain older versions of the package were not longer supported and had >> removed them from PyPI altogether. >> >> This broke the build procedure for my project: I had carefully pinned >> down that I wanted this version (as that's what we tested with), and it >> wasn't available anymore. >> >> So, the removal of old packages from PyPI means that other people who >> rely on these packages with their installation instructions ("install >> Foo 3.3") or build tools (installation instructions, automated), >> suddenly have to deal with breakage. >> >> I thought perhaps PyPI can help and handle this problem at the source. >> >> What are the possible use cases for removing a release from PyPI? >> >> a) mistakes: the upload was accidental - we weren't supposed to release >> this at all! I.e. I uploaded private code that I never meant to >> distribute in the first place. >> >> b) actively hostile: the upload actually contains deliberately malicious >> code and protecting people against this *should* break their builds >> >> c) legal issues: some party claims that this code is not PyPI's to >> distribute in the first place, and for legal issues it'd need to be >> taken down. >> >> d) broken: security or other bugs that makes us want to loudly say, this >> is so broken, don't use it anymore >> >> e) cleaning up old stale unsupported stuff. >> >> I'd argue d) and e) are not up to the package maintainer to decide but >> to the person who integrates this package into their system. The person >> who integrates the package is the one who will need to make the judgment >> call to continue to use old unsupported or broken stuff. Integrators >> should be allowed to make such decisions in their own time at their own >> convenience; the package developer shouldn't be able to force such >> decisions by removing an old release. >> >> (We can think about better warning mechanisms: perhaps there could be >> some metadata or rule to decide when an old release is "deprecated" and >> then integration and installation tools could warn about this, but all >> this would be to help the integrator a better decision on what to do.) >> >> Use cases a), b) and c) are left. I think b) and c) should be handled by >> the PyPI administrators: this takes a judgment call and the package >> maintainer might not be involved in this at all. >> >> Use case a) is the tricky one. I could upload something by mistake, and >> then immediately discover it and decide to throw it out again. But I'd >> be fine if I weren't allowed removing a release anymore after a period >> of a month. It could be that I only discover my mistake after a month, >> when people have already started relying on this version, but in that >> case as a PyPI user I'd want the administrators to issue a judgement >> call there too. >> >> So, my proposal would be to allow people to remove a recent release >> freely, but after a period of (I don't know? A day? A week? A month?) >> removing the old release is not allowed anymore. > > First I have to second Martijn in all aspects (bascially because > are coming from the same projects and facing similar problems). > > The main (meta) question is: is PyPI a the wild-west of the Python world > where everyone can do with packages or do we want PyPI being a well-managed > place for packages with a certain number of rules. Looking at the RSS feed > each day makes me angry sometimes because PyPI feels like the dumpster of > the Python world. The points Martijn raised are only one aspect of this > question. > > -aj > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -- 'Knowledge is Power' Daniel Greenfeld http://pydanny.blogspot.com http://cartwheelweb.com From martin at v.loewis.de Tue Jul 5 19:46:10 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 19:46:10 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> Message-ID: <4E134DE2.2090704@v.loewis.de> > This leads me to conclude that I personally think you have not the > same kind of experience with this issue as I do... I think you haven't > been using packages on a scale I have been doing for years now. Perhaps. I typically use packages from Debian, and try to avoid downloading software from PyPI (or sourceforge or any other place where I need to investigate time to understand how the author does maintenance of hist software). I worry about upgrades every time Debian makes an upgrade, which is fairly infrequent. > Anyway, you're just telling me very loudly I don't have a problem, > right? Not at all. I can see the problem you are having. I'm just suggesting that you should take an entirely different approach to solving it, instead of the one you are heading to (i.e. make the download place provide packages "forever"). Regards, Martin From martin at v.loewis.de Tue Jul 5 19:51:07 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 19:51:07 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E131EE4.2010700@zopyx.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> Message-ID: <4E134F0B.4040007@v.loewis.de> > The main (meta) question is: is PyPI a the wild-west of the Python world > where everyone can do with packages Not sure what picture you have from the wild-west - in my (movie-taught) understanding, the picture would imply that anybody can do anything with *other* people's packages. That is certainly not the case for PyPI. Instead, PyPI is rather the small-town of the Python world: eeverybody can get a parcel of land, and then do on this land basically whatever they want. Business can open and close whenever they please. > or do we want PyPI being a well-managed place I think it's clear that PyPI won't ever be "managed", in any sense of the word, except to make sure that you can't stomp on somebody else's land (i.e. there is self-managed access control). Regards, Martin From martin at v.loewis.de Tue Jul 5 19:52:55 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 19:52:55 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAOoSJ_roGq4Dy1qSU445s_hKwt5g9gRuCFNFscpUOv2n6Wf2QQ@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <CAOoSJ_roGq4Dy1qSU445s_hKwt5g9gRuCFNFscpUOv2n6Wf2QQ@mail.gmail.com> Message-ID: <4E134F77.7040006@v.loewis.de> Am 05.07.2011 19:33, schrieb Daniel Greenfeld: > If that wasn't the original intent > during the inception of PyPI, then maybe the intent ought to be > changed to match this new intention. PyPI is certainly a place for user, i.e. where users can find Python software. The entire user interface is designed for that goal, as are the APIs. Regards, Martin From faassen at startifact.com Tue Jul 5 19:53:30 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 19:53:30 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E134DE2.2090704@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> Message-ID: <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 7:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Not at all. I can see the problem you are having. I'm just suggesting > that you should take an entirely different approach to solving it, > instead of the one you are heading to (i.e. make the download place > provide packages "forever"). Heading to? I've been doing this stuff since 2007. :) In a typical project I'm consuming 20 to a 200 packages, of which I'm maintaining myself perhaps 5 to 20%. The stuff I use typically isn't in Debian or has a version in Debian I don't want to use. It's working pretty well, but this was a small snag I had recently, and I thought, hey, let's see how to improve things. This is my secondary response a few weeks after the event: the first was of course to fix my project and to contact the author. I now think a special mirror might be the simplest way to solve this, since mirroring infrastructure already exist. Regards, Martijn From martin at v.loewis.de Tue Jul 5 20:00:40 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:00:40 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <iuvb3p$js1$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.g mail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> Message-ID: <4E135148.7020204@v.loewis.de> > What's the proportional benefit for those who put stuff in the system? > How much effort is involved in putting things into the system? How would > immutability hinder the benefit? TO give a specific example: I'm getting tired of people asking for PyXML support, despite all pages where they could possibly get it from stating that it's deprecated and unmaintained. At times, I think the best solution would be to completely delete the software. Regards, Martin From faassen at startifact.com Tue Jul 5 19:57:00 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 19:57:00 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E134F0B.4040007@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> Message-ID: <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 7:51 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Instead, PyPI is rather the small-town of the Python world: eeverybody > can get a parcel of land, and then do on this land basically whatever > they want. Business can open and close whenever they please. Sounds like a hosting site for developers for me. The use case "everybody can do whatever they want" conflicts somewhat with the use case of being a system for others to access programmatically for package reuse. Anyway, to us city folk PyPI needs a few more guarantees; things that work in a small town don't always work in the city. >> or do we want PyPI being a well-managed place > > I think it's clear that PyPI won't ever be "managed", in any sense of > the word, except to make sure that you can't stomp on somebody else's > land (i.e. there is self-managed access control). What about making sure there's a historical record? No management built-into the system, but there is history. Regards, Martijn From martin at v.loewis.de Tue Jul 5 20:02:13 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:02:13 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E132147.2040104@zopyx.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E132147.2040104@zopyx.com> Message-ID: <4E1351A5.4060005@v.loewis.de> > Aren't the PyPI mirrors (based on top of the pep381 client) > immutable? No, they synchronize (including deletions). b is different since it runs on appengine, is broken, and I hadn't found the time to fix it yet. Regards, Martin From martin at v.loewis.de Tue Jul 5 20:03:09 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:03:09 +0200 Subject: [Catalog-sig] b mirror out of date? In-Reply-To: <iuvdhr$4ff$1@dough.gmane.org> References: <iuvdhr$4ff$1@dough.gmane.org> Message-ID: <4E1351DD.1050403@v.loewis.de> > Is this intentional? No, it's not. Nobody has found the time to fix it, yet. It runs on appengine, so it uses a different code base. Regards, Martin From martin at v.loewis.de Tue Jul 5 20:10:39 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:10:39 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> Message-ID: <4E13539F.20904@v.loewis.de> Am 05.07.2011 19:53, schrieb Martijn Faassen: > Hi there, > > On Tue, Jul 5, 2011 at 7:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Not at all. I can see the problem you are having. I'm just suggesting >> that you should take an entirely different approach to solving it, >> instead of the one you are heading to (i.e. make the download place >> provide packages "forever"). > > Heading to? I've been doing this stuff since 2007. :) I think you misunderstood: I was talking about the approach you are heading to (i.e. a download place that provides packages "forever"). ... unless I misunderstood, and you have been using such a mirror since 2007. > The stuff I use typically isn't in Debian or > has a version in Debian I don't want to use. I try to avoid software that is not packaged in Debian, and work around limitations that the versions in Debian may have. Regards, Martin From martin at v.loewis.de Tue Jul 5 20:11:28 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:11:28 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> Message-ID: <4E1353D0.70001@v.loewis.de> > What about making sure there's a historical record? No management > built-into the system, but there is history. There *is* a historical record, indeed. Regards, Martin From pydanny at gmail.com Tue Jul 5 20:12:12 2011 From: pydanny at gmail.com (Daniel Greenfeld) Date: Tue, 5 Jul 2011 11:12:12 -0700 Subject: [Catalog-sig] b mirror out of date? In-Reply-To: <4E1351DD.1050403@v.loewis.de> References: <iuvdhr$4ff$1@dough.gmane.org> <4E1351DD.1050403@v.loewis.de> Message-ID: <CAOoSJ_o33V71Jdr3SCGq2E4pPgAf3oUi2cOM6+REawCAwh4XOQ@mail.gmail.com> If it is so far out of date and no one has time to fix it, then maybe it should be taken off the mirror list? Danny On Tue, Jul 5, 2011 at 11:03 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Is this intentional? > > No, it's not. Nobody has found the time to fix it, yet. > It runs on appengine, so it uses a different code base. > > Regards, > Martin > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- 'Knowledge is Power' Daniel Greenfeld http://pydanny.blogspot.com http://cartwheelweb.com From faassen at startifact.com Tue Jul 5 20:24:21 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:24:21 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E135148.7020204@v.loewis.de> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> Message-ID: <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:00 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> What's the proportional benefit for those who put stuff in the system? >> How much effort is involved in putting things into the system? How would >> immutability hinder the benefit? > > TO give a specific example: I'm getting tired of people asking for PyXML > support, despite all pages where they could possibly get it from stating > that it's deprecated and unmaintained. At times, I think the best > solution would be to completely delete the software. An interesting example. I'm not convinced deleting PyXML would stop these people. :) We're talking about proportional benefit here, though. This sounds like a very rare case. It seems that "not wanting to support software" is somehow connected to "not providing the software for download" in people's minds - this is exactly the reason why the maintainer of the package that triggered this discussion for me removed these older versions. And as a result he probably got more support issues than if if he'd left them around. :) You can also see this as a historical record or version control issue. Just because version 0.7 is in a version control system somewhere doesn't mean that the developers are going to support this version. But in this case, I don't see developers argue there that this historical version should therefore be removed. With dependencies, my project in version control (or in released tarball form) has a dependency on an external system. I could instead check all dependencies into my version control system, or I could depend on an external system with more guarantees. That's the feedback I'm getting: either use a system with more guarantees such as a Debian release, or check everything into my own version control system. But this isn't always feasible. * debian might not have my dependencies available yet, or a different version, * I want to hack on the stable and development version of my project without having to install a virtual linux for each to make sure the debian dependencies are right * I want to automate the installation procedure, and therefore can't write an INSTALL.txt which says "apt-get this" * I want have a cross distribution compatible installation instructions Checking stuff into my own version control system has less issues and is indeed workable. There is a paradox there though: I want to release my package to PyPI in a reusable form and therefore can't include all Python dependencies inside it. So I'm not relying on PyPI to download packages myself but I'm asking consumers of my package to do so? Regards, Martijn From faassen at startifact.com Tue Jul 5 20:26:28 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:26:28 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E13539F.20904@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> Message-ID: <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:10 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Am 05.07.2011 19:53, schrieb Martijn Faassen: >> On Tue, Jul 5, 2011 at 7:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>> Not at all. I can see the problem you are having. I'm just suggesting >>> that you should take an entirely different approach to solving it, >>> instead of the one you are heading to (i.e. make the download place >>> provide packages "forever"). >> >> Heading to? I've been doing this stuff since 2007. :) > > I think you misunderstood: I was talking about the approach you are > heading to (i.e. a download place that provides packages "forever"). > ... unless I misunderstood, and you have been using such a mirror > since 2007. You're suggesting I completely rework the approach I have been using since 2007 to avoid breakage due to people removing releases. >> The stuff I use typically isn't in Debian or >> has a version in Debian I don't want to use. > > I try to avoid software that is not packaged in Debian, and > work around limitations that the versions in Debian may have. This works for a variety of use cases. But not for a whole bunch of other use cases. For instance: I need to deploy on a non-Debian platform. Now you again. :) Regards, Martijn From martin at v.loewis.de Tue Jul 5 20:26:40 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:26:40 +0200 Subject: [Catalog-sig] b mirror out of date? In-Reply-To: <CAOoSJ_o33V71Jdr3SCGq2E4pPgAf3oUi2cOM6+REawCAwh4XOQ@mail.gmail.com> References: <iuvdhr$4ff$1@dough.gmane.org> <4E1351DD.1050403@v.loewis.de> <CAOoSJ_o33V71Jdr3SCGq2E4pPgAf3oUi2cOM6+REawCAwh4XOQ@mail.gmail.com> Message-ID: <4E135760.4060207@v.loewis.de> Am 05.07.2011 20:12, schrieb Daniel Greenfeld: > If it is so far out of date and no one has time to fix it, then maybe > it should be taken off the mirror list? That shouldn't be necessary. Mirror clients should use the last-modified time stamp to find out that it is out of date. Regards, Martin From faassen at startifact.com Tue Jul 5 20:27:11 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:27:11 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E1353D0.70001@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> <4E1353D0.70001@v.loewis.de> Message-ID: <CAGT4ZFibgiv9DqWNTc4_MgfxP=9uAYSQL5-u9M5EDpYNRSRj1Q@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:11 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> What about making sure there's a historical record? No management >> built-into the system, but there is history. > > There *is* a historical record, indeed. Really? Where can I find this record? I.e. if you remove release 1.2 of foo, where can I find it again? Regards, Martijn From martin at v.loewis.de Tue Jul 5 20:42:30 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:42:30 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> Message-ID: <4E135B16.5070805@v.loewis.de> > It seems that "not wanting to support software" is somehow connected > to "not providing the software for download" in people's minds Indeed. As a package author, I find that highly plausible. If the software is no longer available anymore, people will stop using it, so they won't bother me with support requests. > this > is exactly the reason why the maintainer of the package that triggered > this discussion for me removed these older versions. And as a result > he probably got more support issues than if if he'd left them around. > :) That's only temporary. When the dust settles, people with either move to a newer version, or use something else altogether. > You can also see this as a historical record or version control issue. > Just because version 0.7 is in a version control system somewhere > doesn't mean that the developers are going to support this version. In free software, you have to support stuff when people contact you. So removal is the easiest way to reduce the amount of time you have to spend for support. > But in this case, I don't see developers argue there that this > historical version should therefore be removed. That's because the users that cause support effort don't check out the software from the repository - those that do check out old versions don't contact you for support. > With dependencies, my project in version control (or in released > tarball form) has a dependency on an external system. I could instead > check all dependencies into my version control system, or I could > depend on an external system with more guarantees. For example, you could depend on the source control system of the software you are depending on. > That's the feedback > I'm getting: either use a system with more guarantees such as a Debian > release, or check everything into my own version control system. But > this isn't always feasible. I can understand the problem. I'm just telling you that the solution you propose is an unacceptable interference with the freedom of the package authors (and this comes from somebody who thinks that a rating system is *not* an unacceptable interference with that freedom). > * debian might not have my dependencies available yet, or a different version, I won't give advice to you, just saying what I do: use the version in Debian. If it doesn't work, adjust your code until it does. If something isn't in Debian, I try to work without the package, possibly rewriting some of the functionality. If the amount of rewriting would be too large, I make an exception and install from PyPI. When doing so, I try to find a version whose dependencies can again be satisfied from PyPI. > * I want to hack on the stable and development version of my project > without having to install a virtual linux for each to make sure the > debian dependencies are right I try to depend only on released version of software, both for stable and development versions, and preferably on the same version of the dependency. So I can use a single Python installation. Regards, Martin From martin at v.loewis.de Tue Jul 5 20:44:16 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:44:16 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFibgiv9DqWNTc4_MgfxP=9uAYSQL5-u9M5EDpYNRSRj1Q@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> <4E1353D0.70001@v.loewis.de> <CAGT4ZFibgiv9DqWNTc4_MgfxP=9uAYSQL5-u9M5EDpYNRSRj1Q@mail.gmail.com> Message-ID: <4E135B80.8010904@v.loewis.de> Am 05.07.2011 20:27, schrieb Martijn Faassen: > Hi there, > > On Tue, Jul 5, 2011 at 8:11 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>> What about making sure there's a historical record? No management >>> built-into the system, but there is history. >> >> There *is* a historical record, indeed. > > Really? Where can I find this record? I.e. if you remove release 1.2 > of foo, where can I find it again? You can't find the release, but you can find a record of the date when the release was deleted. I can also find out who deleted it; this is not published, though. Regards, Martin From jim at zope.com Tue Jul 5 20:45:52 2011 From: jim at zope.com (Jim Fulton) Date: Tue, 5 Jul 2011 14:45:52 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> Message-ID: <CAPDm-FjJiY6Ok8mX2T5NhK9htsk=cDOWnwuM27WCYFZ1TFqERQ@mail.gmail.com> On Tue, Jul 5, 2011 at 1:53 PM, Martijn Faassen <faassen at startifact.com> wrote: ... > I now think a special mirror might be the simplest way to solve this, > since mirroring infrastructure already exist. Of course, it would have to be *special*, in that it would mirror additions and updates, but not removal. Also, ironically, this hurts developers because it makes it harder for them to remove things that *really* need to be removed. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From martin at v.loewis.de Tue Jul 5 20:46:43 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Jul 2011 20:46:43 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> Message-ID: <4E135C13.3010207@v.loewis.de> > On Tue, Jul 5, 2011 at 8:10 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Am 05.07.2011 19:53, schrieb Martijn Faassen: >>> On Tue, Jul 5, 2011 at 7:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>>> Not at all. I can see the problem you are having. I'm just suggesting >>>> that you should take an entirely different approach to solving it, >>>> instead of the one you are heading to (i.e. make the download place >>>> provide packages "forever"). >>> >>> Heading to? I've been doing this stuff since 2007. :) >> >> I think you misunderstood: I was talking about the approach you are >> heading to (i.e. a download place that provides packages "forever"). >> ... unless I misunderstood, and you have been using such a mirror >> since 2007. I take that back. I'm not proposing that you change anything about the way you work. Instead, I'm just reporting that there are completely alternative approaches possible at all which don't have the problem you are trying to solve. Regards, Martin From faassen at startifact.com Tue Jul 5 20:49:10 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:49:10 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E135B80.8010904@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E131EE4.2010700@zopyx.com> <4E134F0B.4040007@v.loewis.de> <CAGT4ZFiZSn0UbSE60fQX7uL3KTk4HuYKyteW2qSBYW09-ReV3A@mail.gmail.com> <4E1353D0.70001@v.loewis.de> <CAGT4ZFibgiv9DqWNTc4_MgfxP=9uAYSQL5-u9M5EDpYNRSRj1Q@mail.gmail.com> <4E135B80.8010904@v.loewis.de> Message-ID: <CAGT4ZFgB1ivtgTj7KNFneLZQr_3+P=JDCdjq4fmFSgBNkKv18Q@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:44 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Am 05.07.2011 20:27, schrieb Martijn Faassen: >> On Tue, Jul 5, 2011 at 8:11 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>>> What about making sure there's a historical record? No management >>>> built-into the system, but there is history. >>> >>> There *is* a historical record, indeed. >> >> Really? Where can I find this record? I.e. if you remove release 1.2 >> of foo, where can I find it again? > > You can't find the release, but you can find a record of the date when > the release was deleted. I can also find out who deleted it; this is > not published, though. Indeed a history of the index, not of the things being indexed, even if they're in PyPI itself. Regards, Martijn From faassen at startifact.com Tue Jul 5 20:51:35 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:51:35 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <4E135C13.3010207@v.loewis.de> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> <4E135C13.3010207@v.loewis.de> Message-ID: <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:46 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > I take that back. I'm not proposing that you change anything about > the way you work. Instead, I'm just reporting that there are completely > alternative approaches possible at all which don't have the problem you > are trying to solve. Sure. I was wrong to rely on PyPI to download and install the packages I need. Debian it is! :) Regards, Martijn From faassen at startifact.com Tue Jul 5 20:56:06 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 5 Jul 2011 20:56:06 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E135B16.5070805@v.loewis.de> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> Message-ID: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Hi there, On Tue, Jul 5, 2011 at 8:42 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: [snip] > I can understand the problem. I'm just telling you that the solution > you propose is an unacceptable interference with the freedom of the > package authors (and this comes from somebody who thinks that a rating > system is *not* an unacceptable interference with that freedom). Sure. Unacceptable interference with the freedom of package authors to remove stuff. Unacceptable because developers need the freedom to remove old versions of their software from the internet. :) So if someone comes up with a immutable mirror, do you think that's unacceptable interference with the freedom of package authors as well? Just checking. I'll just switch to debian for my package management needs. :) Regards, Martijn From ziade.tarek at gmail.com Tue Jul 5 23:50:30 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 5 Jul 2011 23:50:30 +0200 Subject: [Catalog-sig] b mirror out of date? In-Reply-To: <CAGSi+Q7Uqkap9jj=Q7mB+WLswjJnknWUfwSJhAR332CfJm8Ftw@mail.gmail.com> References: <iuvdhr$4ff$1@dough.gmane.org> <4E1351DD.1050403@v.loewis.de> <CAOoSJ_o33V71Jdr3SCGq2E4pPgAf3oUi2cOM6+REawCAwh4XOQ@mail.gmail.com> <4E135760.4060207@v.loewis.de> <CAGSi+Q7Uqkap9jj=Q7mB+WLswjJnknWUfwSJhAR332CfJm8Ftw@mail.gmail.com> Message-ID: <CAGSi+Q5OsiN5_+uHpSvvnYhTmY+QhT22FLpyY00gzM3c6siUXg@mail.gmail.com> Oops forgot to reply all Le 5 juil. 2011 23:48, "Tarek Ziad?" <ziade.tarek at gmail.com> a ?crit : > They should but some people are still picking a mirror by hand. I think a > good policy could be to remove mirrors which lag > 1 month > > Tarek > Le 5 juil. 2011 20:27, Martin v. L?wis <martin at v.loewis.de> a ?crit : >> Am 05.07.2011 20:12, schrieb Daniel Greenfeld: >>> If it is so far out of date and no one has time to fix it, then maybe >>> it should be taken off the mirror list? >> >> That shouldn't be necessary. Mirror clients should use the last-modified >> time stamp to find out that it is out of date. >> >> Regards, >> Martin >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110705/54cf67fb/attachment.html> From richard at python.org Tue Jul 5 23:55:54 2011 From: richard at python.org (Richard Jones) Date: Wed, 6 Jul 2011 07:55:54 +1000 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> <4E135C13.3010207@v.loewis.de> <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> Message-ID: <CAHrZfZB+DHR1LQmFmsgCyD0_b21hyQRdmDsG8YFiKTrcqwqMDw@mail.gmail.com> Could we rewind right back to the start of this conversation when it was suggested that you keep and ship a local mirror of packages you rely on? Such mirroring is trivial to set up. Richard From pje at telecommunity.com Wed Jul 6 03:19:27 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 05 Jul 2011 21:19:27 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.g mail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Message-ID: <20110706011938.0B6C13A4108@sparrow.telecommunity.com> At 08:56 PM 7/5/2011 +0200, Martijn Faassen wrote: >Sure. Unacceptable interference with the freedom of package authors to >remove stuff. Unacceptable because developers need >the freedom to remove old versions of their software from the internet. :) You still seem to be ignoring the part where insisting that people keep old things around for your convenience *also* means they get support requests from everyone *else* on the internet. In other words, a package author has to pay an (ongoing and perpetual) support cost for your temporary convenience. You are not making a principled moral stand here. Rather, you are asking that other people be made to carry an additional cost, for your benefit, on the basis that you don't think their cost is important, but you think your benefit is. From pje at telecommunity.com Wed Jul 6 03:23:11 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 05 Jul 2011 21:23:11 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAPDm-FjJiY6Ok8mX2T5NhK9htsk=cDOWnwuM27WCYFZ1TFqERQ@mail.g mail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <CAPDm-FjJiY6Ok8mX2T5NhK9htsk=cDOWnwuM27WCYFZ1TFqERQ@mail.gmail.com> Message-ID: <20110706012317.04C003A4108@sparrow.telecommunity.com> At 02:45 PM 7/5/2011 -0400, Jim Fulton wrote: >On Tue, Jul 5, 2011 at 1:53 PM, Martijn Faassen ><faassen at startifact.com> wrote: >... > > I now think a special mirror might be the simplest way to solve this, > > since mirroring infrastructure already exist. > >Of course, it would have to be *special*, in that it would mirror >additions and updates, but not removal. > >Also, ironically, this hurts developers because it makes it harder for >them to remove things that *really* need to be removed. I think it's well understood that you can't put the internet back in the bottle. But disallowing removals in the first place takes away the package author's right to decide what "really" means. In any case, the developer is no worse off than in the case where they have to convince the PyPI operators something "really" needs to be removed, and they are better off because at least the official distribution point for their package no longer carries the broken release. From lists at zopyx.com Wed Jul 6 06:20:03 2011 From: lists at zopyx.com (Andreas Jung) Date: Wed, 06 Jul 2011 06:20:03 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110706011938.0B6C13A4108@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <20110706011938.0B6C13A4108@sparrow.telecommunity.com> Message-ID: <4E13E273.8040803@zopyx.com> P.J. Eby wrote: > At 08:56 PM 7/5/2011 +0200, Martijn Faassen wrote: >> Sure. Unacceptable interference with the freedom of package authors to >> remove stuff. Unacceptable because developers need >> the freedom to remove old versions of their software from the >> internet. :) > > You still seem to be ignoring the part where insisting that people keep > old things around for your convenience *also* means they get support > requests from everyone *else* on the internet. In other words, a package > author has to pay an (ongoing and perpetual) support cost for your > temporary convenience. > > You are not making a principled moral stand here. Rather, you are asking > that other people be made to carry an additional cost, for your benefit, > on the basis that you don't think their cost is important, but you think > your benefit is. As a package author using PyPI for spreading my work for myself _and_ for other people implies that I take over a certain responsibility for - package quality and maturity - availability of my packages - reasonable metadata If a package author won't stick with those principles then a general hint to the author would be: stay away from PyPI and drop your package crap somewhere else. I don't care if other people see it different but PyPI must not be a place for autistic programmers that don't look left and right to the needs of package users and don't care about PyPI and general and that consider PyPI only their own private package sink. -aj -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110706/cd36eb73/attachment.vcf> From lists at zopyx.com Wed Jul 6 06:41:55 2011 From: lists at zopyx.com (Andreas Jung) Date: Wed, 06 Jul 2011 06:41:55 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E135B16.5070805@v.loewis.de> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> Message-ID: <4E13E793.5090904@zopyx.com> Martin v. L?wis wrote: > > I can understand the problem. I'm just telling you that the solution > you propose is an unacceptable interference with the freedom of the > package authors (and this comes from somebody who thinks that a rating > system is *not* an unacceptable interference with that freedom). Freedom of authors is one side of the medal. Playing nice in a community and sticking to some standard when using a community resource like PyPI is the other side. Freedom of authors does not stand over the needs of the community. Super-programmers not interested accepting certain rules still can do their own thing while leaving the community out of the game. -aj -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110706/e2f35c72/attachment.vcf> From martin at v.loewis.de Wed Jul 6 09:00:01 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Jul 2011 09:00:01 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Message-ID: <4E1407F1.5010105@v.loewis.de> > So if someone comes up with a immutable mirror, do you think that's > unacceptable interference with the freedom of package authors as well? I'm sure some package authors will object to such an installation. For the packages I maintain personally, doing so will be fine - I just don't want to be in charge of such a mirror, or have to defend it. Regards, Martin From mailing at franzoni.eu Wed Jul 6 09:59:16 2011 From: mailing at franzoni.eu (Alan Franzoni) Date: Wed, 6 Jul 2011 09:59:16 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Message-ID: <CAF3z5=mng+oocU0DTCDgA3r+r9tpRB_tmxHjn__e43hOsGnU-g@mail.gmail.com> On Tue, Jul 5, 2011 at 8:56 PM, Martijn Faassen <faassen at startifact.com> wrote: [cut] I had some thoughts myself on the topic; I think the idea of an immutable (or almost-immutable) pypi would be a rather good one. If you've had some kind of experience with Maven repositories in the Java world you know they're usually a good idea for dependency management; then you can setup your own internal proxy/mirror which you can rely on if some external server fades away. Mirroring a mutable pypi is not as easy as it sounds: what do I do if a release file is deleted and replaced with another file, same release number? How should my mirror work? Maven repos usually never allow for artifacts to be removed from their interface - you should remove them at filesystem level, but then you know you're doing something the system doesn't want you to. Rubygems approach is different but still takes care of version immutability a bit: you can yank a release, thus removing the artifact, but you're prevented from reuploading anything for the same release. The downside of mutability is: - today, I've got a working project which fetches dependencies through setuptools; - the project stays the same, one year passes. - The project no longer works because some deps on pypi changed or were removed. There're of course some good use cases for package removal, but they're "edgy": e.g. leaked credentials in source code, leaked unauthorized code, etc. OTOH there's the maintenance problem for package authors - I'd suggest something like a way to deprecate packages and releases, so that when anybody fetches such packages it can get a "deprecated" warning and should know the author is not maintaining such code anymore, but won't prevent legitimate and informed users to happily keep going with their existing software. Of course all of those things that I propose might just not fit into PyPI. If anybody is interested in creating a PyRepo or something like that, just let me know and I'd happily collaborate. -- http://www.franzoni.eu - public@[mysurname].eu Latest blog post: Unit testing with Twisted: testing protocols: http://t.co/HFpslG4 From mal at egenix.com Wed Jul 6 11:10:01 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Jul 2011 11:10:01 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> Message-ID: <4E142669.7090307@egenix.com> Martijn Faassen wrote: > Hi there, > > On Tue, Jul 5, 2011 at 8:42 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > [snip] >> I can understand the problem. I'm just telling you that the solution >> you propose is an unacceptable interference with the freedom of the >> package authors (and this comes from somebody who thinks that a rating >> system is *not* an unacceptable interference with that freedom). > > Sure. Unacceptable interference with the freedom of package authors to > remove stuff. Unacceptable because developers need > the freedom to remove old versions of their software from the internet. :) > > So if someone comes up with a immutable mirror, do you think that's > unacceptable interference with the freedom of package authors as well? > Just checking. There are situations where you have to be able to remove a package file or even a complete package and by introducing public immutable servers you make it harder for package authors or PyPI maintainers to resolve such issues. Some possible reasons: * legal action (copyright, trademark, DMCA, license issues, etc) * removal of malicious packages (e.g. script kiddy stuff in setup.py) * reassigning package names (not sure whether that's possible with PyPI, but it certainly happens in the wild every now and then) * renaming packages (e.g. due to a poor initial package name choice) * seriously broken builds (e.g. that cause users to lose data) Private immutable index servers should not be a problem, since the package author or the PSF (as legal entity behind PyPI) has no direct way of fixing those. Please note that PyPI is getting bigger every day, so even if we are not seeing many such cases now, the problems can arise and if they do, they are going to cause a lot of trouble. Even as owner of a private immutable mirror, you would still want to know about such issues and I'm not sure whether you're really up to dealing with those cases as mirror maintainer. I'm repeating myself, but I still think you're better off creating a special mirror for your projects which then only has the packages and versions you want and need in your projects. Setting up such a mirror is really easy. All it takes is some web server, a directory with the package files and a script to create a static PyPI-style simple index file.... but you know all that, after all, Zope and Plone have been running such PyPI-style repositories for years :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 06 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mailing at franzoni.eu Wed Jul 6 16:20:52 2011 From: mailing at franzoni.eu (Alan Franzoni) Date: Wed, 6 Jul 2011 16:20:52 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E142669.7090307@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> Message-ID: <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> On Wed, Jul 6, 2011 at 11:10 AM, M.-A. Lemburg <mal at egenix.com> wrote: > Some possible reasons: > ?* renaming packages (e.g. due to a poor initial package name > ? choice) This is not a good reason, IMHO. You can go on with new versions and a new name, maybe you could want to deprecate the old package, but it's not a good reason to remove it. > ?* legal action (copyright, trademark, DMCA, license issues, etc) > > ?* removal of malicious packages (e.g. script kiddy stuff in > ? setup.py) > ?* seriously broken builds (e.g. that cause users to lose data) This is would make a good reason for package removal, but not for version reassignment. I.E. if I delete version 0.4.3 because it deleted my /usr instead of /usr/share/mylib/content, then I would be right at removing it, but there's no point in allowing any other package to take back 0.4.3. It's simply gone. > ?* reassigning package names (not sure whether that's possible with > ? PyPI, but it certainly happens in the wild every now and then) I'm not sure about what you mean here. BTW I think that the main issue here is *why* anybody should delete anything from pypi. If your conditions above apply, it's perfectly right for a dev to remove the packages. But I think most packages are removed without such conditions. Since I think such events are pretty rare, there could be a ticketing system dedicated to an "emergency removal" of artifacts, and that reupload of an artifact with the very same name-release should be inhibited. -- http://www.franzoni.eu - public@[mysurname].eu Latest blog post: Unit testing with Twisted: testing protocols: http://t.co/HFpslG4 From mal at egenix.com Wed Jul 6 17:54:04 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Jul 2011 17:54:04 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> Message-ID: <4E14851C.6070402@egenix.com> Please note that I'm talking about the idea of an immutable mirror, not PyPI itself - unfortunately, you've cut away that important context from my email. Alan Franzoni wrote: > On Wed, Jul 6, 2011 at 11:10 AM, M.-A. Lemburg <mal at egenix.com> wrote: >> Some possible reasons: > >> * renaming packages (e.g. due to a poor initial package name >> choice) > > This is not a good reason, IMHO. You can go on with new versions and a > new name, maybe you could want to deprecate the old package, but it's > not a good reason to remove it. I think undoing mistakes in package names is a very good reason to remove them. As package author you don't want such mistakes to stay on the net forever, if you can avoid it. >> * legal action (copyright, trademark, DMCA, license issues, etc) >> >> * removal of malicious packages (e.g. script kiddy stuff in >> setup.py) >> * seriously broken builds (e.g. that cause users to lose data) > > This is would make a good reason for package removal, but not for > version reassignment. I.E. if I delete version 0.4.3 because it > deleted my /usr instead of /usr/share/mylib/content, then I would be > right at removing it, but there's no point in allowing any other > package to take back 0.4.3. It's simply gone. I'm not sure I understand what you want to say. I wasn't talking about version reassignment in the above cases. >> * reassigning package names (not sure whether that's possible with >> PyPI, but it certainly happens in the wild every now and then) > > I'm not sure about what you mean here. Author A releases a package X, then drops the idea and removes the package, freeing up the name for others to use. Later on, author B uses the name X for something different and creates a new package X with a new set of releases. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 06 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From michael at voidspace.org.uk Fri Jul 8 21:22:40 2011 From: michael at voidspace.org.uk (Michael Foord) Date: Fri, 08 Jul 2011 20:22:40 +0100 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> Message-ID: <4E175900.6070907@voidspace.org.uk> Would it be possible to configure pypi to not recommend contacting webmaster at python.org <mailto:webmaster at python.org> on server error, but contact the pypi admins instead? All the best, Michael Foord -------- Original Message -------- Subject: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ Date: Fri, 8 Jul 2011 15:11:09 -0400 From: Jordan Goldstein <jordango at mit.edu> To: webmaster at python.org Enjoy. Please try again later. The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, webmaster at python.org <mailto:webmaster at python.org> and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. ------------------------------------------------------------------------ Apache/2.2.16 (Debian) Server at pypi.python.org <http://pypi.python.org> Port 80 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110708/8b8cf4d7/attachment.html> From sergey at maluke.com Sat Jul 9 17:14:02 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Sat, 9 Jul 2011 18:14:02 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi Message-ID: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> Hi, So there was some talk in paste-users list on releasing beta versions on PyPi or not, as easy_install and others pick them up by default. ( https://groups.google.com/d/msg/paste-users/1UAuadQjKng/qxVuF6lNU6QJ ) I've also had problems caused by that before (random segfaults due to beta release of lxml), so I can relate to the problem, but I still think betas should go to pypi. So, I've hacked together what seems like an obvious fix -- a pypi mirror/proxy that only includes stable versions. http://pypi-stable.maluke.com/ Anyway, just thought some people in this list might be interested in this. For more info see the thread linked above. -Sergey -- http://self.maluke.com/ From lists at zopyx.com Sun Jul 10 08:01:10 2011 From: lists at zopyx.com (Andreas Jung) Date: Sun, 10 Jul 2011 08:01:10 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> Message-ID: <4E194026.9040002@zopyx.com> Sorry but a pretty worthless effort since there is no enforced version schema and in addition the classifiers are often just unmaintained or wrong. -aj Sergey Schetinin wrote: > Hi, > > So there was some talk in paste-users list on releasing beta versions > on PyPi or not, as easy_install and others pick them up by default. ( > https://groups.google.com/d/msg/paste-users/1UAuadQjKng/qxVuF6lNU6QJ ) > I've also had problems caused by that before (random segfaults due to > beta release of lxml), so I can relate to the problem, but I still > think betas should go to pypi. > > So, I've hacked together what seems like an obvious fix -- a pypi > mirror/proxy that only includes stable versions. > http://pypi-stable.maluke.com/ > > Anyway, just thought some people in this list might be interested in this. > For more info see the thread linked above. > > -Sergey > -- ZOPYX Limited | zopyx group Charlottenstr. 37/1 | The full-service network for Zope & Plone D-72070 T?bingen | Produce & Publish www.zopyx.com | www.produce-and-publish.com ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110710/fb0bfae9/attachment.vcf> From ben+python at benfinney.id.au Sun Jul 10 09:10:08 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 10 Jul 2011 17:10:08 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> Message-ID: <87mxgmfw6n.fsf@benfinney.id.au> Andreas Jung <lists at zopyx.com> writes: > Sorry but a pretty worthless effort since there is no enforced version > schema and in addition the classifiers are often just unmaintained or > wrong. Conversely, the presence of a tool that uses the classifier information in this way could encourage people to put useful information in there. +1. I'm in favour, all else being equal, of having a distinct repository of stable-release-only packages. -- \ ?The Vatican is not a state.? a state must have people. There | `\ are no Vaticanians.? No-one gets born in the Vatican except by | _o__) an unfortunate accident.? ?Geoffrey Robertson, 2010-09-18 | Ben Finney From ziade.tarek at gmail.com Sun Jul 10 10:11:20 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 10 Jul 2011 10:11:20 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E194026.9040002@zopyx.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> Message-ID: <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> This is not true anymore for metadata 1.2 which follow pep 386. Pypi already implements it. Pushing a non-386 version gets your action rejected. Plus distutils 2 has a prefer final flag Cheers Tarek Le 10 juil. 2011 08:01, "Andreas Jung" <lists at zopyx.com> a ?crit : > Sorry but a pretty worthless effort since there is no enforced version > schema and in addition the classifiers are often just unmaintained or wrong. > > -aj > > Sergey Schetinin wrote: >> Hi, >> >> So there was some talk in paste-users list on releasing beta versions >> on PyPi or not, as easy_install and others pick them up by default. ( >> https://groups.google.com/d/msg/paste-users/1UAuadQjKng/qxVuF6lNU6QJ ) >> I've also had problems caused by that before (random segfaults due to >> beta release of lxml), so I can relate to the problem, but I still >> think betas should go to pypi. >> >> So, I've hacked together what seems like an obvious fix -- a pypi >> mirror/proxy that only includes stable versions. >> http://pypi-stable.maluke.com/ >> >> Anyway, just thought some people in this list might be interested in this. >> For more info see the thread linked above. >> >> -Sergey >> > > -- > ZOPYX Limited | zopyx group > Charlottenstr. 37/1 | The full-service network for Zope & Plone > D-72070 T?bingen | Produce & Publish > www.zopyx.com | www.produce-and-publish.com > ------------------------------------------------------------------------ > E-Publishing, Python, Zope & Plone development, Consulting > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110710/2cd26b80/attachment.html> From lists at zopyx.com Sun Jul 10 11:39:28 2011 From: lists at zopyx.com (Andreas Jung) Date: Sun, 10 Jul 2011 11:39:28 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> Message-ID: <4E197350.6040204@zopyx.com> Tarek Ziad? wrote: > This is not true anymore for metadata 1.2 which follow pep 386. Which is likely a very small number of packages. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 310 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110710/f23ad579/attachment.vcf> From sergey at maluke.com Sun Jul 10 16:24:12 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Sun, 10 Jul 2011 17:24:12 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E194026.9040002@zopyx.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> Message-ID: <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.gmail.com> On 10 July 2011 09:01, Andreas Jung <lists at zopyx.com> wrote: > Sorry but a pretty worthless effort since there is no enforced version > schema and in addition the classifiers are often just unmaintained or wrong. Sure. All it does is fix the problem right now, without telling anyone to change how they do releases etc. Worthless. If you know of any releases that are final but are filtered out in that mirror, that would be more constructive. -- http://self.maluke.com/ From pje at telecommunity.com Sun Jul 10 17:51:05 2011 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 10 Jul 2011 11:51:05 -0400 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.g mail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.gmail.com> Message-ID: <20110710155132.3C9243A409B@sparrow.telecommunity.com> At 05:24 PM 7/10/2011 +0300, Sergey Schetinin wrote: >If you know of any releases that are final but are filtered out in >that mirror, that would be more constructive. ISTM that you are currently filtering out packages whose *only* available versions are alpha, beta, candidate, etc. So if a package is still experimental or in development, it is not available. You might want to consider using a "most stable available release type" instead. That is, if a project only has alphas and betas, provide the betas. From sergey at maluke.com Sun Jul 10 17:53:38 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Sun, 10 Jul 2011 18:53:38 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <20110710155132.3C9243A409B@sparrow.telecommunity.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.gmail.com> <20110710155132.3C9243A409B@sparrow.telecommunity.com> Message-ID: <CAP2iL+C4Rw3ymYtqG6f7P0-WReBSGDB+zBHwVU5hUbmmQa5PBg@mail.gmail.com> On 10 July 2011 18:51, P.J. Eby <pje at telecommunity.com> wrote: > At 05:24 PM 7/10/2011 +0300, Sergey Schetinin wrote: >> >> If you know of any releases that are final but are filtered out in >> that mirror, that would be more constructive. > > ISTM that you are currently filtering out packages whose *only* available > versions are alpha, beta, candidate, etc. ?So if a package is still > experimental or in development, it is not available. > > You might want to consider using a "most stable available release type" > instead. ?That is, if a project only has alphas and betas, provide the > betas. Good point, thanks. -- http://self.maluke.com/ From sergey at maluke.com Sun Jul 10 18:31:43 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Sun, 10 Jul 2011 19:31:43 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+C4Rw3ymYtqG6f7P0-WReBSGDB+zBHwVU5hUbmmQa5PBg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAP2iL+D2Tnw0kS8fvW787G6ZxNe2HVjoPynQHnuJTV2tN9nGJQ@mail.gmail.com> <20110710155132.3C9243A409B@sparrow.telecommunity.com> <CAP2iL+C4Rw3ymYtqG6f7P0-WReBSGDB+zBHwVU5hUbmmQa5PBg@mail.gmail.com> Message-ID: <CAP2iL+CU+=B1cge3Ofrc1Rtn8M5u2La7RcVZOcqKE3vzUDOcAQ@mail.gmail.com> So I did what Phillip suggested. Also I now don't filter out the less-stable releases completely -- I remove the links, but still list them, so it's easier to see what the proxy does. For example: http://pypi-stable.maluke.com/simple/setuptools/ http://pypi-stable.maluke.com/simple/lxml http://pypi-stable.maluke.com/simple/WebOb -- http://self.maluke.com/ From merwok at netwok.org Mon Jul 11 14:41:18 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 11 Jul 2011 14:41:18 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> Message-ID: <4E1AEF6E.9020700@netwok.org> Le 10/07/2011 10:11, Tarek Ziad? a ?crit : > This is not true anymore for metadata 1.2 which follow pep 386. Pypi already > implements it. Pushing a non-386 version gets your action rejected. I was under the impression that PEP 386 only defined the syntax of version numbers and a comparison algo, but no semantics. IOW there is no way for a tool to know that 2.6.33 is devel and 2.6.34 stable, or that 1.0.4 does not break compatibility with 1.0.2, or anything else of the sort. > Plus distutils 2 has a prefer final flag Is final the same thing as stable? Regards From tseaver at palladion.com Mon Jul 11 18:45:16 2011 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 11 Jul 2011 12:45:16 -0400 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1AEF6E.9020700@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> Message-ID: <ivf9aq$bgv$1@dough.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/11/2011 08:41 AM, ?ric Araujo wrote: > Le 10/07/2011 10:11, Tarek Ziad? a ?crit : >> This is not true anymore for metadata 1.2 which follow pep 386. >> Pypi already implements it. Pushing a non-386 version gets your >> action rejected. > > I was under the impression that PEP 386 only defined the syntax of > version numbers and a comparison algo, but no semantics. IOW there > is no way for a tool to know that 2.6.33 is devel and 2.6.34 stable, > or that 1.0.4 does not break compatibility with 1.0.2, or anything > else of the sort. The PEP386 semantics are implied by the sorting order, where "final" (non-suffixed) releases sort higher than any suffixed releases except "post" releases. Any suffix from the 'a|b|c|rc' group implies a "non-final" / "development" version of some kind (alpha, beta, release candidate), while suffixes in the 'post' group imply some kind of patch-level (presumed stable unless re-suffixed with 'dev'). Note that zc.buildout already has a 'prefer-final' option which uses these semantics: http://pypi.python.org/pypi/zc.buildout/1.5.2#preferring-final-releases Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4bKJwACgkQ+gerLs4ltQ4M4QCeK7p3jzqzuWl3VzZJJ8gyw7+L IM8AoLgr/1+l3tmbpBVBwF9nJKYt+QkD =K3oO -----END PGP SIGNATURE----- From ziade.tarek at gmail.com Mon Jul 11 19:42:16 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 19:42:16 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E197350.6040204@zopyx.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> Message-ID: <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> On Sun, Jul 10, 2011 at 11:39 AM, Andreas Jung <lists at zopyx.com> wrote: > > > Tarek Ziad? wrote: >> >> This is not true anymore for metadata 1.2 which follow pep 386. > > Which is likely a very small number of packages. Oh yeah, maybe none except mine right now :) but the point is: I think there's some value right now in adding PEP 386 support in existing tools + a prefer-final option in every installers, than trying to set up a new 'stable only' pypi It turns out that around 80% of PyPI packages are PEP 386 compliant already.. > > Andreas > -- Tarek Ziad? | http://ziade.org From sergey at maluke.com Mon Jul 11 20:33:45 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Mon, 11 Jul 2011 21:33:45 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> Message-ID: <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> On 11 July 2011 20:42, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > but the point is: I think there's some value right now in adding PEP > 386 support in existing tools + a prefer-final option in every > installers, than trying to set up a new 'stable only' pypi Well, if the data is there, it's super-easy to filter on the server-side. And why repeat that work in every installer if it can be accomplished by supplying a different pypi index url? -- http://self.maluke.com/ From ziade.tarek at gmail.com Mon Jul 11 20:43:57 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 20:43:57 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> Message-ID: <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> On Mon, Jul 11, 2011 at 8:33 PM, Sergey Schetinin <sergey at maluke.com> wrote: > On 11 July 2011 20:42, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >> but the point is: I think there's some value right now in adding PEP >> 386 support in existing tools + a prefer-final option in every >> installers, than trying to set up a new 'stable only' pypi > > Well, if the data is there, it's super-easy to filter on the > server-side. Is that a proxy project for you or do you want the community to create a new pypi server that filters out non-stable releases ? If it's the latter, what version scheme will you use to decide whether a version is alpha beta etc. ? It would make sense to use PEP 386. If so, the filter that you are creating can be hooked on client side as easily as server side, the only difference is that you don't have to change the existing infrastructure and deal with mirrors. (unless the proxy is to be run on client side ?) > And why repeat that work in every installer if it can be > accomplished by supplying a different pypi index url? Because ideally we would want all installers to behave the same way wrt versions and PyPI repositories. If I run a mirror of PyPI and I don't use your proxy, I don't have the "prefer final" version. If this is a client-side option it may work on any PyPI repo, mirror, or even on any directory containing packages (since pip and setuptools allow scanning local dirs) > > > -- > http://self.maluke.com/ > -- Tarek Ziad? | http://ziade.org From sergey at maluke.com Mon Jul 11 21:03:57 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Mon, 11 Jul 2011 22:03:57 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> Message-ID: <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> On 11 July 2011 21:43, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > On Mon, Jul 11, 2011 at 8:33 PM, Sergey Schetinin <sergey at maluke.com> wrote: >> On 11 July 2011 20:42, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >>> but the point is: I think there's some value right now in adding PEP >>> 386 support in existing tools + a prefer-final option in every >>> installers, than trying to set up a new 'stable only' pypi >> >> Well, if the data is there, it's super-easy to filter on the >> server-side. > > Is that a proxy project for you or do you want the community to create > a new pypi server that filters out non-stable releases ? > > If it's the latter, what version scheme will you use to decide whether > a version is alpha beta etc. ? ?It would make sense to use PEP 386. If > so, the filter that you are creating can be hooked on client side as > easily as server side, the only difference is that you don't have to > change the existing infrastructure and deal with mirrors. > > (unless the proxy is to be run on client side ?) That webapp (it's not *really* a proxy) took me about two hours total, and it has to query pypi, guess what is stable and not, cache the results. If there's no guessing and all data is local, how hard can it be? I'm not asking anyone to do it, but it does look like it would be a simpler, more immediate solution. My point is, the server can use whatever the most current metadata format is out there to make the decision, maybe falling back to guessing if the package does not include it, and it would work immediately, with every install tool that supports custom package indices. However I see that the mirrors issue makes this impractical. -- http://self.maluke.com/ From ziade.tarek at gmail.com Mon Jul 11 21:20:36 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 21:20:36 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> Message-ID: <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: ... > > That webapp (it's not *really* a proxy) took me about two hours total, > and it has to query pypi, guess what is stable and not, cache the > results. how do you guess it ? that's an interesting part > If there's no guessing and all data is local, how hard can it > be? I'm not asking anyone to do it, but it does look like it would be > a simpler, more immediate solution. istm that it can be complex to have it working over the mirrors, unless it's a proxy on client-side. But then we're closer to a client-side change I guess. > > My point is, the server can use whatever the most current metadata > format is out there to make the decision, maybe falling back to > guessing if the package does not in nclude it, and it would work > immediately, with every install tool that supports custom package > indices. So right now the 'version' field is used only -- and present in all metadata versions at PyPI. There are now three versions schemes out there to my knowledge, with small differences (see PEP 386 for the details) pep 386 (#3) is still irrelevant but will start to gain traction once 3.3 is out. Maybe there's a way to sort stable vs none stable versions across all schemes and versions.. > However I see that the mirrors issue makes this impractical. Yes that's the main concern > > -- > http://self.maluke.com/ > -- Tarek Ziad? | http://ziade.org From sergey at maluke.com Mon Jul 11 21:32:36 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Mon, 11 Jul 2011 22:32:36 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> Message-ID: <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> On 11 July 2011 22:20, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: > ... >> >> That webapp (it's not *really* a proxy) took me about two hours total, >> and it has to query pypi, guess what is stable and not, cache the >> results. > > how do you guess it ? that's an interesting part Well, I mentioned it in linked post. Basically at first I was doing just this: _rx_final = re.compile(r'[\d\.]+\Z') def is_final(ver): return bool(_rx_final.match(ver)) But to implement the most-stable filter, I changed it to this: http://pastebin.com/JVVAJ9hE >> If there's no guessing and all data is local, how hard can it >> be? I'm not asking anyone to do it, but it does look like it would be >> a simpler, more immediate solution. > > istm that it can be complex to have it working over the mirrors, > unless it's a proxy on client-side. > But then we're closer to a client-side change I guess. Given the existing mirrors infrastructure it is indeed hard. If the mirrors were demoted to just serving downloads and not the index, they could still be used with such an index as-is. As I said, I have to agree it would be impractical to add a stable-only pypi index and try to mirror it as well. -- http://self.maluke.com/ From ziade.tarek at gmail.com Mon Jul 11 21:47:29 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 21:47:29 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> Message-ID: <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> On Mon, Jul 11, 2011 at 9:32 PM, Sergey Schetinin <sergey at maluke.com> wrote: > On 11 July 2011 22:20, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >> On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: >> ... >>> >>> That webapp (it's not *really* a proxy) took me about two hours total, >>> and it has to query pypi, guess what is stable and not, cache the >>> results. >> >> how do you guess it ? that's an interesting part > > Well, I mentioned it in linked post. Basically at first I was doing just this: > > _rx_final = re.compile(r'[\d\.]+\Z') > > def is_final(ver): > ? ?return bool(_rx_final.match(ver)) > > But to implement the most-stable filter, I changed it to this: > http://pastebin.com/JVVAJ9hE there must be a bug because all the versions I am trying are marked stable (even 'lalala') > >>> If there's no guessing and all data is local, how hard can it >>> be? I'm not asking anyone to do it, but it does look like it would be >>> a simpler, more immediate solution. >> >> istm that it can be complex to have it working over the mirrors, >> unless it's a proxy on client-side. >> But then we're closer to a client-side change I guess. > > Given the existing mirrors infrastructure it is indeed hard. If the > mirrors were demoted to just serving downloads and not the index, they > could still be used with such an index as-is. Yeah, the mirror is standalone or it breaks its intent (being useful when PyPI is down) > As I said, I have to > agree it would be impractical to add a stable-only pypi index and try > to mirror it as well. > > > > -- > http://self.maluke.com/ > -- Tarek Ziad? | http://ziade.org From sergey at maluke.com Mon Jul 11 21:56:44 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Mon, 11 Jul 2011 22:56:44 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> Message-ID: <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> On 11 July 2011 22:47, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > On Mon, Jul 11, 2011 at 9:32 PM, Sergey Schetinin <sergey at maluke.com> wrote: >> On 11 July 2011 22:20, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >>> On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: >>> ... >>>> >>>> That webapp (it's not *really* a proxy) took me about two hours total, >>>> and it has to query pypi, guess what is stable and not, cache the >>>> results. >>> >>> how do you guess it ? that's an interesting part >> >> Well, I mentioned it in linked post. Basically at first I was doing just this: >> >> _rx_final = re.compile(r'[\d\.]+\Z') >> >> def is_final(ver): >> ? ?return bool(_rx_final.match(ver)) >> >> But to implement the most-stable filter, I changed it to this: >> http://pastebin.com/JVVAJ9hE > > there must be a bug because all the versions I am trying are marked stable > (even 'lalala') Not a bug. It marks the most stable versions available: >>> mark_stable(['lala']) [('lala', True)] >>> mark_stable(['lala', '1.0']) [('lala', False), ('1.0', True)] >>> mark_stable(['0.1a1']) [('0.1a1', True)] >>> mark_stable(['0.1a1', '0.1b1']) [('0.1a1', False), ('0.1b1', True)] >>> mark_stable(['0.1a1', '0.1b1', '0.1']) [('0.1a1', False), ('0.1b1', False), ('0.1', True)] >>>> If there's no guessing and all data is local, how hard can it >>>> be? I'm not asking anyone to do it, but it does look like it would be >>>> a simpler, more immediate solution. >>> >>> istm that it can be complex to have it working over the mirrors, >>> unless it's a proxy on client-side. >>> But then we're closer to a client-side change I guess. >> >> Given the existing mirrors infrastructure it is indeed hard. If the >> mirrors were demoted to just serving downloads and not the index, they >> could still be used with such an index as-is. > > Yeah, the mirror is standalone or it breaks its intent (being useful > when PyPI is down) It's unfortunate this instability was accepted for granted. -- http://self.maluke.com/ From ziade.tarek at gmail.com Mon Jul 11 22:21:47 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 11 Jul 2011 22:21:47 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> Message-ID: <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> On Mon, Jul 11, 2011 at 9:56 PM, Sergey Schetinin <sergey at maluke.com> wrote: > On 11 July 2011 22:47, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >> On Mon, Jul 11, 2011 at 9:32 PM, Sergey Schetinin <sergey at maluke.com> wrote: >>> On 11 July 2011 22:20, Tarek Ziad? <ziade.tarek at gmail.com> wrote: >>>> On Mon, Jul 11, 2011 at 9:03 PM, Sergey Schetinin <sergey at maluke.com> wrote: >>>> ... >>>>> >>>>> That webapp (it's not *really* a proxy) took me about two hours total, >>>>> and it has to query pypi, guess what is stable and not, cache the >>>>> results. >>>> >>>> how do you guess it ? that's an interesting part >>> >>> Well, I mentioned it in linked post. Basically at first I was doing just this: >>> >>> _rx_final = re.compile(r'[\d\.]+\Z') >>> >>> def is_final(ver): >>> ? ?return bool(_rx_final.match(ver)) >>> >>> But to implement the most-stable filter, I changed it to this: >>> http://pastebin.com/JVVAJ9hE >> >> there must be a bug because all the versions I am trying are marked stable >> (even 'lalala') > > Not a bug. It marks the most stable versions available: > >>>> mark_stable(['lala']) > [('lala', True)] >>>> mark_stable(['lala', '1.0']) > [('lala', False), ('1.0', True)] >>>> mark_stable(['0.1a1']) > [('0.1a1', True)] >>>> mark_stable(['0.1a1', '0.1b1']) > [('0.1a1', False), ('0.1b1', True)] >>>> mark_stable(['0.1a1', '0.1b1', '0.1']) > [('0.1a1', False), ('0.1b1', False), ('0.1', True)] ah ok :D ... >> >> Yeah, the mirror is standalone or it breaks its intent (being useful >> when PyPI is down) > > It's unfortunate this instability was accepted for granted. Yeah, well we've made a lot of progress with PEP 386 / Metadata 1.2 Right now you can still push projects with a version number being "bazinga-boomya". Try to sort this.. PyPI will reject non-pep 386 versions with the newest metadata 1.2, that's why I am pushing hard on its adoption on client-side because it's a better long term solution. Having to guess if a version is stable or not with a version scheme that's free-form is quite a pain. The short-term fix that works well is to ask people not to push unstable stuff at pypi... people can always pick unstable version on the project website Cheers Tarek -- Tarek Ziad? | http://ziade.org From sdouche at gmail.com Tue Jul 12 02:45:36 2011 From: sdouche at gmail.com (Sebastien Douche) Date: Tue, 12 Jul 2011 02:45:36 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> Message-ID: <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> On Mon, Jul 11, 2011 at 22:21, Tarek Ziad? <ziade.tarek at gmail.com> wrote: Hi Tarek > PyPI will reject non-pep 386 versions with the newest metadata 1.2, > that's why I am pushing hard on its adoption on client-side because > it's a better long term solution. Right > Having to guess if a version is stable or not with a version scheme > that's free-form is quite a pain. The short-term fix that works well > is to ask people not to push unstable stuff at pypi... people can > always pick unstable version on the project website Or add a new stable index on PyPI (ex: http://python.org/pypi/stable). Easy to create, easy to maintain, easy to use and don't break anything. -- Sebastien Douche <sdouche at gmail.com> Twitter : @sdouche From martin at v.loewis.de Tue Jul 12 08:14:14 2011 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 12 Jul 2011 08:14:14 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> Message-ID: <4E1BE636.4040904@v.loewis.de> > Or add a new stable index on PyPI (ex: http://python.org/pypi/stable). > Easy to create, easy to maintain, easy to use and don't break > anything. If people can agree on a specification as to what packages this should include exactly, I can happily provide it. I won't lead a project to get people agree, though. If you go by trove classifiers, my proposal would be to include all releases with a classifier "Development Status :: 5 - Production/Stable" or "Development Status :: 6 - Mature". Regards, Martin From sergey at maluke.com Tue Jul 12 08:17:39 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Tue, 12 Jul 2011 09:17:39 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1BE636.4040904@v.loewis.de> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> Message-ID: <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> On 12 July 2011 09:14, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Or add a new stable index on PyPI (ex: http://python.org/pypi/stable). >> Easy to create, easy to maintain, easy to use and don't break >> anything. > > If people can agree on a specification as to what packages this should > include exactly, I can happily provide it. I won't lead a project to > get people agree, though. > > If you go by trove classifiers, my proposal would be to include all > releases with a classifier "Development Status :: 5 - Production/Stable" > or "Development Status :: 6 - Mature". That's very unlikely to work. Once webob got to maturity we set that classifier and it's there even for repo snapshots. I'm sure it's the case for ~100% other projects. -- http://self.maluke.com/ From g.brandl at gmx.net Tue Jul 12 08:28:55 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 12 Jul 2011 08:28:55 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> Message-ID: <ivgpi0$bok$1@dough.gmane.org> Am 12.07.2011 08:17, schrieb Sergey Schetinin: > On 12 July 2011 09:14, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>> Or add a new stable index on PyPI (ex: http://python.org/pypi/stable). >>> Easy to create, easy to maintain, easy to use and don't break >>> anything. >> >> If people can agree on a specification as to what packages this should >> include exactly, I can happily provide it. I won't lead a project to >> get people agree, though. >> >> If you go by trove classifiers, my proposal would be to include all >> releases with a classifier "Development Status :: 5 - Production/Stable" >> or "Development Status :: 6 - Mature". > > That's very unlikely to work. Once webob got to maturity we set that > classifier and it's there even for repo snapshots. I'm sure it's the > case for ~100% other projects. Yeah, the classifier seems to be more appropriate for the project's status, not that of individual releases. (Otherwise it would be a pain to remember to reset that classifier for every prerelease, when all it does is repeat one bit of information already present in the version number.) Georg From ben+python at benfinney.id.au Tue Jul 12 08:37:03 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 12 Jul 2011 16:37:03 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> Message-ID: <87oc10dmy8.fsf@benfinney.id.au> "Martin v. L?wis" <martin at v.loewis.de> writes: > If you go by trove classifiers, my proposal would be to include all > releases with a classifier "Development Status :: 5 - Production/Stable" > or "Development Status :: 6 - Mature". Yes, I think trove classifiers are the right data to use for filtering, and not version string. The ?Development Status? trove category is *exactly* where this information should be found, and corrected if not there. -- \ ?I busted a mirror and got seven years bad luck, but my lawyer | `\ thinks he can get me five.? ?Steven Wright | _o__) | Ben Finney From ben+python at benfinney.id.au Tue Jul 12 08:38:21 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 12 Jul 2011 16:38:21 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> Message-ID: <87k4bodmw2.fsf@benfinney.id.au> Georg Brandl <g.brandl at gmx.net> writes: > (Otherwise it would be a pain to remember to reset that classifier for > every prerelease, when all it does is repeat one bit of information > already present in the version number.) Why is the version string repeating information that is more precisely represented in the trove category? I'd say that's where the problem lies. -- \ ?For every complex problem, there is a solution that is simple, | `\ neat, and wrong.? ?Henry L. Mencken | _o__) | Ben Finney From martin at v.loewis.de Tue Jul 12 08:44:44 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 12 Jul 2011 08:44:44 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> Message-ID: <4E1BED5C.3080006@v.loewis.de> > That's very unlikely to work. Once webob got to maturity we set that > classifier and it's there even for repo snapshots. I'm sure it's the > case for ~100% other projects. It seems you are right, although there are certainly counter examples, such as http://pypi.python.org/pypi/zw.schema/0.2.2 http://pypi.python.org/pypi/zw.schema/0.3.0b2.1 In any case, as I said, I'll stay out of any attempt to specify something here. Regards, Martin From sergey at maluke.com Tue Jul 12 08:47:02 2011 From: sergey at maluke.com (Sergey Schetinin) Date: Tue, 12 Jul 2011 09:47:02 +0300 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <87oc10dmy8.fsf@benfinney.id.au> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <87oc10dmy8.fsf@benfinney.id.au> Message-ID: <CAP2iL+DpYce3zo_2zvEUnmwpiX6yE24oKT5MMA4wfiD9pVSxYw@mail.gmail.com> On 12 July 2011 09:37, Ben Finney <ben+python at benfinney.id.au> wrote: > "Martin v. L?wis" <martin at v.loewis.de> writes: > >> If you go by trove classifiers, my proposal would be to include all >> releases with a classifier "Development Status :: 5 - Production/Stable" >> or "Development Status :: 6 - Mature". > > Yes, I think trove classifiers are the right data to use for filtering, > and not version string. > > The ?Development Status? trove category is *exactly* where this > information should be found, and corrected if not there. This filtering has to work on as many existing releases as possible, not just the ones made from this point, not even on the most recent ones, it should work with what's actually in pypi. -- http://self.maluke.com/ From ben+python at benfinney.id.au Tue Jul 12 09:12:00 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 12 Jul 2011 17:12:00 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <87oc10dmy8.fsf@benfinney.id.au> <CAP2iL+DpYce3zo_2zvEUnmwpiX6yE24oKT5MMA4wfiD9pVSxYw@mail.gmail.com> Message-ID: <87fwmcdlbz.fsf@benfinney.id.au> Sergey Schetinin <sergey at maluke.com> writes: > On 12 July 2011 09:37, Ben Finney <ben+python at benfinney.id.au> wrote: > > The ?Development Status? trove category is *exactly* where this > > information should be found, and corrected if not there. > > This filtering has to work on as many existing releases as possible, > not just the ones made from this point, not even on the most recent > ones, it should work with what's actually in pypi. Well, if the proposal involves living with bad data and not fixing it, I withdraw my approval (for whatever that was worth). -- \ ?When I was a kid I used to pray every night for a new bicycle. | `\ Then I realised that the Lord doesn't work that way so I stole | _o__) one and asked Him to forgive me.? ?Emo Philips | Ben Finney From g.brandl at gmx.net Tue Jul 12 09:18:38 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 12 Jul 2011 09:18:38 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <87k4bodmw2.fsf@benfinney.id.au> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> <87k4bodmw2.fsf@benfinney.id.au> Message-ID: <ivgsf7$pkl$1@dough.gmane.org> Am 12.07.2011 08:38, schrieb Ben Finney: > Georg Brandl <g.brandl at gmx.net> writes: > >> (Otherwise it would be a pain to remember to reset that classifier for >> every prerelease, when all it does is repeat one bit of information >> already present in the version number.) > > Why is the version string repeating information that is more precisely > represented in the trove category? I'd say that's where the problem > lies. Sorry, I don't understand. Are you suggesting calling both 1.0 alpha 1 and 1.0 final "version 1.0" and just changing the trove classifier? Georg From merwok at netwok.org Tue Jul 12 15:14:22 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 12 Jul 2011 15:14:22 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <ivgpi0$bok$1@dough.gmane.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> Message-ID: <4E1C48AE.4060409@netwok.org> Le 12/07/2011 08:28, Georg Brandl a ?crit : > Am 12.07.2011 08:17, schrieb Sergey Schetinin: >> On 12 July 2011 09:14, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>> If you go by trove classifiers, my proposal would be to include all >>> releases with a classifier "Development Status :: 5 - Production/Stable" >>> or "Development Status :: 6 - Mature". >> >> That's very unlikely to work. Once webob got to maturity we set that >> classifier and it's there even for repo snapshots. I'm sure it's the >> case for ~100% other projects. > > Yeah, the classifier seems to be more appropriate for the project's status, > not that of individual releases. ISTM that the discussion is confusing ?stable? and ?final?. In my understanding, stable = project status = Trove classifier final = release status = PEP 386 version Regards From merwok at netwok.org Tue Jul 12 15:25:34 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 12 Jul 2011 15:25:34 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <ivf9aq$bgv$1@dough.gmane.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> Message-ID: <4E1C4B4E.7030107@netwok.org> Le 11/07/2011 18:45, Tres Seaver a ?crit : > On 07/11/2011 08:41 AM, ?ric Araujo wrote: >> I was under the impression that PEP 386 only defined the syntax of >> version numbers and a comparison algo, but no semantics. IOW there >> is no way for a tool to know that 2.6.33 is devel and 2.6.34 stable, >> or that 1.0.4 does not break compatibility with 1.0.2, or anything >> else of the sort. > > The PEP386 semantics are implied by the sorting order, where "final" > (non-suffixed) releases sort higher than any suffixed releases except > "post" releases. I?m aware of the algo for sorting and asserting final/non-final. I was talking about stability and compatibility policies: PEP 386, contrary to the misnamed Semantic Versioning spec for example, does not define anything in this regard (this was already pointed out a few months ago by PJE, I think). See the examples in the quoted text above. Regards From tseaver at palladion.com Tue Jul 12 15:59:07 2011 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jul 2011 09:59:07 -0400 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1C4B4E.7030107@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> Message-ID: <ivhjvb$9rd$1@dough.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/12/2011 09:25 AM, ?ric Araujo wrote: > Le 11/07/2011 18:45, Tres Seaver a ?crit : >> On 07/11/2011 08:41 AM, ?ric Araujo wrote: >>> I was under the impression that PEP 386 only defined the syntax >>> of version numbers and a comparison algo, but no semantics. IOW >>> there is no way for a tool to know that 2.6.33 is devel and >>> 2.6.34 stable, or that 1.0.4 does not break compatibility with >>> 1.0.2, or anything else of the sort. >> >> The PEP386 semantics are implied by the sorting order, where >> "final" (non-suffixed) releases sort higher than any suffixed >> releases except "post" releases. > > I?m aware of the algo for sorting and asserting final/non-final. I > was talking about stability and compatibility policies: PEP 386, > contrary to the misnamed Semantic Versioning spec for example, does > not define anything in this regard (this was already pointed out a > few months ago by PJE, I think). See the examples in the quoted > text above. Hmm? In what sense would 2.6.33 ever be used for a "devel" release? If my application got broken by a project that made such a release, I would be busy ripping out a dependency produced by such an unreliable project, not arguing whether PyPI could / should implement some "technical measure" to make everything happy. If your point is that a "stable-only" mirror could still induce breakage on projects which use it blindly, I certainly concur. In fact, I have long asserted that any "integrator" whose production builds uses PyPI directly is solely responsible for the breakage, when (not if) it occurs. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4cUysACgkQ+gerLs4ltQ5kQwCg1Ed7iv0/1ag1NxN07vqCVWbl IAsAoM/1xY5BPu/4AG0aCQdxLhQnEQEh =I+yw -----END PGP SIGNATURE----- From merwok at netwok.org Tue Jul 12 16:26:45 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 12 Jul 2011 16:26:45 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <ivhjvb$9rd$1@dough.gmane.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> Message-ID: <4E1C59A5.4020406@netwok.org> > Hmm? In what sense would 2.6.33 ever be used for a "devel" release? By the linux kernel project, for example: http://en.wikipedia.org/wiki/Software_versioning#Odd-numbered_versions_for_development_releases > If my application got broken by a project that made such a release, I > would be busy ripping out a dependency produced by such an unreliable > project, not arguing whether PyPI could / should implement some > "technical measure" to make everything happy. Granted, this linux kernel versioning example was extreme. However, compatibility issues between X.Y.Z and X.Y.Z+1 is not unheard of, and that?s a problem that PEP 386 does not address. (Even if it did, nothing would prevent human error or malice, we?d still need human checks, but at least we?d have a common base for expectations.) > If your point is that a "stable-only" mirror could still induce breakage > on projects which use it blindly, I certainly concur. My point is that ?stability? is not defined, at least not by PEP 386. And even when the Trove classifiers are set, blindly trusting them would not be a good idea. Regards From ziade.tarek at gmail.com Tue Jul 12 16:44:57 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 12 Jul 2011 16:44:57 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1C59A5.4020406@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> <4E1C59A5.4020406@netwok.org> Message-ID: <CAGSi+Q54-M9vXDFTedbwaHR2VW9F8jR9wfcPR-yiom4qgcfHiA@mail.gmail.com> On Tue, Jul 12, 2011 at 4:26 PM, ?ric Araujo <merwok at netwok.org> wrote: >> Hmm? ?In what sense would 2.6.33 ever be used for a "devel" release? > > By the linux kernel project, for example: > http://en.wikipedia.org/wiki/Software_versioning#Odd-numbered_versions_for_development_releases > >> If my application got broken by a project that made such a release, I >> would be busy ripping out a dependency produced by such an unreliable >> project, not arguing whether PyPI could / should implement some >> "technical measure" to make everything happy. > > Granted, this linux kernel versioning example was extreme. ?However, > compatibility issues between X.Y.Z and X.Y.Z+1 is not unheard of, and > that?s a problem that PEP 386 does not address. ?(Even if it did, > nothing would prevent human error or malice, we?d still need human > checks, but at least we?d have a common base for expectations.) We provide a versionning scheme with a dev tag to be used for dev releases, if people push a dev version under a version number without the dev flag, that's their fault. There's no difference here between a trove classifier and a version scheme. Those are just conventions, flags. So I am not sure how you want to address this issue exactly, a dev marker is a dev marker... and dev/non-dev != stability for backward compat issues, it's a problem to be addressed at the project level, with its own conventions. > >> If your point is that a "stable-only" mirror could still induce breakage >> on projects which use it blindly, I certainly concur. > > My point is that ?stability? is not defined, at least not by PEP 386. You can't define what stability means in a generic way in the versionning scheme. We have dev marker, beta alpa, rc markers but that's just transitional states between dev and prod for the sake of feedback. It's up to every project to define stability, and I agree that the Trove is a good way to do it. > And even when the Trove classifiers are set, blindly trusting them would > not be a good idea. Ah you lost me here. If the project uses a trove classifier to mark "stable", either you trust the project either you don't. Deciding if you upgrade to the newer version in your environment is your own problem and orthogonal > > Regards > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- Tarek Ziad? | http://ziade.org From tseaver at palladion.com Tue Jul 12 16:55:31 2011 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 12 Jul 2011 10:55:31 -0400 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1C59A5.4020406@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> <4E1C59A5.4020406@netwok.org> Message-ID: <ivhn93$u7k$1@dough.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/12/2011 10:26 AM, ?ric Araujo wrote: >> Hmm? In what sense would 2.6.33 ever be used for a "devel" >> release? > > By the linux kernel project, for example: > http://en.wikipedia.org/wiki/Software_versioning#Odd-numbered_versions_for_development_releases That > would have been 2.3.36 vs. 2.2 29: the even / odd variation for development / stable was based on the minor release number, not the third. Also note that the kernel devs dropped that pattern quite a while back as being unfruitful. >> If my application got broken by a project that made such a release, >> I would be busy ripping out a dependency produced by such an >> unreliable project, not arguing whether PyPI could / should >> implement some "technical measure" to make everything happy. > > Granted, this linux kernel versioning example was extreme. However, > compatibility issues between X.Y.Z and X.Y.Z+1 is not unheard of, > and that?s a problem that PEP 386 does not address. (Even if it > did, nothing would prevent human error or malice, we?d still need > human checks, but at least we?d have a common base for > expectations.) We *do* have a common base for expectations: the set of projects on PyPI which violate those expectations (about the semantics of the version number) is pretty small. Backward-[in]compatibility policy is just not a documented part of those expectations: you have to learn each project's policy about BBB issues as part of evaluating whether to use its releases. >> If your point is that a "stable-only" mirror could still induce >> breakage on projects which use it blindly, I certainly concur. > > My point is that ?stability? is not defined, at least not by PEP > 386. And even when the Trove classifiers are set, blindly trusting > them would not be a good idea. Sure: blindly trusting any project / author is not a good idea for people who need highly-reliable / reproducible build-deployment solutions. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4cYGMACgkQ+gerLs4ltQ6ThwCgn4j9GAf2Vm9TR8C9AQO6VLZ+ bE8AoLQsfXMYK0r3buANjMqegAbl8Lkt =xr3/ -----END PGP SIGNATURE----- From g.brandl at gmx.net Tue Jul 12 20:00:44 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 12 Jul 2011 20:00:44 +0200 Subject: [Catalog-sig] Stable-releases-only PyPi In-Reply-To: <4E1C48AE.4060409@netwok.org> References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> <4E1C48AE.4060409@netwok.org> Message-ID: <ivi234$6qf$2@dough.gmane.org> Am 12.07.2011 15:14, schrieb ?ric Araujo: > Le 12/07/2011 08:28, Georg Brandl a ?crit : >> Am 12.07.2011 08:17, schrieb Sergey Schetinin: >>> On 12 July 2011 09:14, "Martin v. L?wis" <martin at v.loewis.de> wrote: >>>> If you go by trove classifiers, my proposal would be to include all >>>> releases with a classifier "Development Status :: 5 - Production/Stable" >>>> or "Development Status :: 6 - Mature". >>> >>> That's very unlikely to work. Once webob got to maturity we set that >>> classifier and it's there even for repo snapshots. I'm sure it's the >>> case for ~100% other projects. >> >> Yeah, the classifier seems to be more appropriate for the project's status, >> not that of individual releases. > > ISTM that the discussion is confusing ?stable? and ?final?. > > In my understanding, > stable = project status = Trove classifier > final = release status = PEP 386 version Exactly. While a project itself can be stable (or mature, even), individual releases still can be of pre-release quality. Georg From ben+python at benfinney.id.au Tue Jul 12 23:43:46 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 13 Jul 2011 07:43:46 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E197350.6040204@zopyx.com> <CAGSi+Q5T0Yaek6fweTzUiaCCg74dvEQU0J1x2+eiAJBjRa=Uzw@mail.gmail.com> <CAP2iL+Dyzq3cOXVmKEX3hUThQ7tiVWiZFUrmCMjLL-i0RvNifQ@mail.gmail.com> <CAGSi+Q556kxSVwD199crOMBNk-mmY6Sb0c+cRvXSoqTSL_sWAw@mail.gmail.com> <CAP2iL+AFWV=dhwmmQ1doW2C=sYOWEsg3QeEq=+5gTeau-roGyQ@mail.gmail.com> <CAGSi+Q74-1+B-GYxA6HYe5DzLqJJObwLrvcSZ+_uZkscig5RgA@mail.gmail.com> <CAP2iL+Cj723LqwssVBqnQdH83ooWEFLD0kUFtsu8RVNzJn5ZTg@mail.gmail.com> <CAGSi+Q4dhZy+Oqc88dnhXmZRmZMBCxnU3bZq4_+hKKvZZRGhog@mail.gmail.com> <CAP2iL+B-vfs57hCXBoaV37cnUZrLS57MYuytZOMvoDQHX5iEWQ@mail.gmail.com> <CAGSi+Q6XeyuFy0BzAJKacq3nDd1D9uqNW1pQZ+wCZ1fFG72KFg@mail.gmail.com> <CAAGHeXFeAUsMfP1c-_53mKfRTxEgb26vttB57oEjg5f42K=BOw@mail.gmail.com> <4E1BE636.4040904@v.loewis.de> <CAP2iL+DVpLKg72-8Q5qQQSL-VFk=Mg65xxydmhH2+VUAof5rkQ@mail.gmail.com> <ivgpi0$bok$1@dough.gmane.org> <4E1C48AE.4060409@netwok.org> Message-ID: <877h7ndvjh.fsf@benfinney.id.au> ?ric Araujo <merwok at netwok.org> writes: > ISTM that the discussion is confusing ?stable? and ?final?. > > In my understanding, > stable = project status = Trove classifier You're right, I'm wrong. The trove classifier is more appropriate for the project status. -- \ ?We can't depend for the long run on distinguishing one | `\ bitstream from another in order to figure out which rules | _o__) apply.? ?Eben Moglen, _Anarchism Triumphant_, 1999 | Ben Finney From ben+python at benfinney.id.au Tue Jul 12 23:55:52 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 13 Jul 2011 07:55:52 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> Message-ID: <87zkkjcgev.fsf@benfinney.id.au> Tres Seaver <tseaver at palladion.com> writes: > On 07/12/2011 09:25 AM, ?ric Araujo wrote: > >> On 07/11/2011 08:41 AM, ?ric Araujo wrote: > >>> I was under the impression that PEP 386 only defined the syntax of > >>> version numbers and a comparison algo, but no semantics. IOW there > >>> is no way for a tool to know that 2.6.33 is devel and 2.6.34 > >>> stable, or that 1.0.4 does not break compatibility with 1.0.2, or > >>> anything else of the sort. [?] > > Hmm? In what sense would 2.6.33 ever be used for a "devel" release? Why should it not be? Projects are not bound to insert words anywhere in their version strings for any state of the code. I find inserting words into a version string to be ugly and overly complicated, and I don't recommend it for any project. Fortunately, conforming with PEP 386 doesn't require anyone to do that. Nor does PEP 386 define what a version string like ?2.6.33? means. It only says how that version string will sort against other version strings. > If my application got broken by a project that made such a release, I > would be busy ripping out a dependency produced by such an unreliable > project, not arguing whether PyPI could / should implement some > "technical measure" to make everything happy. That expectation (that a development version must have a particular word inserted in the version string) is unreasonable and has no foundation in PEP 386. A tool for automatically producing a ?stable-only? mirror can't rely on the version string containing any information about the development status of a version; not even one which conforms perfectly to PEP 386. > If your point is that a "stable-only" mirror could still induce breakage > on projects which use it blindly, I certainly concur. In fact, I have > long asserted that any "integrator" whose production builds uses PyPI > directly is solely responsible for the breakage, when (not if) it occurs. +1. This is what OS distributions are good for: selecting which package versions are suitable for the stability of the OS as a whole. -- \ ?I stayed up all night playing poker with tarot cards. I got a | `\ full house and four people died.? ?Steven Wright | _o__) | Ben Finney -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110713/87ac8f34/attachment.pgp> From ben+python at benfinney.id.au Wed Jul 13 00:02:40 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 13 Jul 2011 08:02:40 +1000 Subject: [Catalog-sig] Stable-releases-only PyPi References: <CAP2iL+B-knYrng=G-j4DOgEgDyc+9Cms1S=c5wvk5fcArnBueQ@mail.gmail.com> <4E194026.9040002@zopyx.com> <CAGSi+Q7G2ATb5-vuGVp2RNTtqHkG8dS0V-29+R=nq7u0Ko_ZJg@mail.gmail.com> <4E1AEF6E.9020700@netwok.org> <ivf9aq$bgv$1@dough.gmane.org> <4E1C4B4E.7030107@netwok.org> <ivhjvb$9rd$1@dough.gmane.org> <4E1C59A5.4020406@netwok.org> <CAGSi+Q54-M9vXDFTedbwaHR2VW9F8jR9wfcPR-yiom4qgcfHiA@mail.gmail.com> Message-ID: <87vcv7cg3j.fsf@benfinney.id.au> Tarek Ziad? <ziade.tarek at gmail.com> writes: > We provide a versionning scheme with a dev tag to be used for dev > releases, if people push a dev version under a version number without > the dev flag, that's their fault. How is it a ?fault?? It's up to the project to decide the semantics of their version strings. If PEP 386 is going to be used as a blunt instrument to force semantics (and I agree with Eric that its poorly-chosen name encourages this), then the fault of that is with those who misunderstand what PEP 386 requires. > for backward compat issues, it's a problem to be addressed at the > project level, with its own conventions. Yes, thank you. Let's stop saying ?fault? if a project's chosen meaning for version strings doesn't require words in version strings. > You can't define what stability means in a generic way in the > versionning scheme. We have dev marker, beta alpa, rc markers but > that's just transitional states between dev and prod for the sake of > feedback. It's also not at all required under PEP 386 for any project to use those markers. > It's up to every project to define stability, and I agree that the > Trove is a good way to do it. Yet there seems to be no Trove classifier for the stability of an individual version, as contrasted with the ?overall stability? of the project. Should we re-purpose that trove classifier to this meaning? Maybe. But how to convince everyone to use it that way? -- \ ?First things first, but not necessarily in that order.? ?The | `\ Doctor, _Doctor Who_ | _o__) | Ben Finney From martin at v.loewis.de Thu Jul 14 21:17:05 2011 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 14 Jul 2011 21:17:05 +0200 Subject: [Catalog-sig] b.pypi up-to-date again Message-ID: <4E1F40B1.9030704@v.loewis.de> I fixed the AppEngine PyPI mirror (b.pypi.python.org), and it's up-to-date again. In case you are interested, here is the list of problems: - integration of download stats broke on 2010-01-17. The implementation creates a Download record every time somebody downloads a file from the mirror, and integrates them once a day into a daily report (deleting the Download objects, and creating a Stats object). It combines all Download records except for the ones of the current day, and deletes the one it combines (in chunks of 100, since integrating all at once would exceed the 30s limit in appengine). Now, for Jan 17, integration completed (i.e. GAE said there weren't any further Download objects left); however, the next day, another Download object for the day showed up, breaking the integration job. I fixed it by post-dating it to the next day that hadn't been integrated. - mirroring broke when a file was deleted on PyPI after the mirror learned of its existence, but before it got mirrored. Mirroring works the way that a) the GAE app asks PyPI to upload something (downloading from the master doesn't work because it exceeds the maximum response size for the url fetcher). b) PyPI asks GAE to create an upload URL, and asks what file to upload c) PyPI uploads the file to the blob store d) GAE tells the GAE app that the upload is complete In the scenario, step c) failed, because the file wasn't there anymore, so PyPI just ignored it. The GAE app's cron job then restarted the upload, which would fail again - and so every 5 minutes since March 2010. I fixed it by having PyPI upload a null-byte file, along with a POST flag telling that the file is to be deleted - restarting the mirroring then failed since accessing the changelog since March took about 20s, exceeding the 5s XML-RPC limit of GAE. I fixed it by introducing another RPC, changed_packages. It then took three days to catch up, but should now stay up-to-date. Regards, Martin From lac at openend.se Thu Jul 14 21:24:53 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 14 Jul 2011 21:24:53 +0200 Subject: [Catalog-sig] b.pypi up-to-date again In-Reply-To: Message from =?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?= <martin@v.loewis.de> of "Thu, 14 Jul 2011 21:17:05 +0200." <4E1F40B1.9030704@v.loewis.de> References: <4E1F40B1.9030704@v.loewis.de> Message-ID: <201107141924.p6EJOrC9022351@theraft.openend.se> In a message of Thu, 14 Jul 2011 21:17:05 +0200, "Martin v. L?wis" writes: >I fixed the AppEngine PyPI mirror (b.pypi.python.org), and it's >up-to-date again. Hurrah! (And thank you.) Laura From tjreedy at udel.edu Thu Jul 14 22:25:11 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 14 Jul 2011 16:25:11 -0400 Subject: [Catalog-sig] b.pypi up-to-date again In-Reply-To: <4E1F40B1.9030704@v.loewis.de> References: <4E1F40B1.9030704@v.loewis.de> Message-ID: <ivnjb7$jka$1@dough.gmane.org> On 7/14/2011 3:17 PM, "Martin v. L?wis" wrote: > In case you are interested, here is the list of problems: Thank you for this also. I learn from such reports. -- Terry Jan Reedy From richard at python.org Fri Jul 15 08:41:13 2011 From: richard at python.org (Richard Jones) Date: Fri, 15 Jul 2011 16:41:13 +1000 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <4E175900.6070907@voidspace.org.uk> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> Message-ID: <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> Sorry for the silence. Currently there's no pypi admins alias, though I think it'd probably be nice to have rather than to have three email addresses hard-coded into the application like we have at the moment :-) On the other hand, in terms of this error page - is it possible to customise it to forward the user to the support tracker (ie. the one in the pypi sidebar) rather than email? Richard On 9 July 2011 05:22, Michael Foord <michael at voidspace.org.uk> wrote: > Would it be possible to configure pypi to not recommend contacting > webmaster at python.org on server error, but contact the pypi admins instead? > > All the best, > > Michael Foord > > -------- Original Message -------- Subject: Recieved following error at > 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ Date: Fri, 8 Jul 2011 > 15:11:09 -0400 From: Jordan Goldstein <jordango at mit.edu><jordango at mit.edu> To: > webmaster at python.org > > Enjoy. > > Please try again later. > > The server encountered an internal error or misconfiguration and was unable > to complete your request. > > Please contact the server administrator, webmaster at python.org and inform > them of the time the error occurred, and anything you might have done that > may have caused the error. > > More information about this error may be available in the server error log. > ------------------------------ > Apache/2.2.16 (Debian) Server at pypi.python.org Port 80 > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110715/5005efce/attachment.html> From michael at voidspace.org.uk Fri Jul 15 13:07:00 2011 From: michael at voidspace.org.uk (Michael Foord) Date: Fri, 15 Jul 2011 12:07:00 +0100 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> Message-ID: <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> On 15 July 2011 07:41, Richard Jones <richard at python.org> wrote: > Sorry for the silence. > > Currently there's no pypi admins alias, though I think it'd probably be > nice to have rather than to have three email addresses hard-coded into the > application like we have at the moment :-) > > On the other hand, in terms of this error page - is it possible to > customise it to forward the user to the support tracker (ie. the one in the > pypi sidebar) rather than email? > > The particular error page is just the standard error 500 page served by Apache - and it is possible to configure Apache to serve a custom one for the pypi subdomain. Not that I know *specifically how* to do that, but I know it is possible. :-) A pypi admin address, that just forwards to this list (?) sounds like a very good idea (as well). All the best, Michael > > Richard > > On 9 July 2011 05:22, Michael Foord <michael at voidspace.org.uk> wrote: > >> Would it be possible to configure pypi to not recommend contacting >> webmaster at python.org on server error, but contact the pypi admins >> instead? >> >> All the best, >> >> Michael Foord >> >> -------- Original Message -------- Subject: Recieved following error at >> 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ Date: Fri, 8 Jul 2011 >> 15:11:09 -0400 From: Jordan Goldstein <jordango at mit.edu><jordango at mit.edu> To: >> webmaster at python.org >> >> Enjoy. >> >> Please try again later. >> >> The server encountered an internal error or misconfiguration and was >> unable to complete your request. >> >> Please contact the server administrator, webmaster at python.org and inform >> them of the time the error occurred, and anything you might have done that >> may have caused the error. >> >> More information about this error may be available in the server error >> log. >> ------------------------------ >> Apache/2.2.16 (Debian) Server at pypi.python.org Port 80 >> >> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig >> >> > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110715/6a88c46b/attachment.html> From richard at python.org Fri Jul 15 13:27:40 2011 From: richard at python.org (Richard Jones) Date: Fri, 15 Jul 2011 21:27:40 +1000 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> Message-ID: <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> On 15 July 2011 21:07, Michael Foord <michael at voidspace.org.uk> wrote: > The particular error page is just the standard error 500 page served by > Apache - and it is possible to configure Apache to serve a custom one for > the pypi subdomain. Not that I know *specifically how* to do that, but I > know it is possible. :-) > Yeah, that's pretty much where I'm at :-) Really, the hard part (because I have no control over it) is setting up the alias. > A pypi admin address, that just forwards to this list (?) sounds like a > very good idea (as well). > I'm not sure this list really wants to be caught up in the administrivia of running pypi :-) Richard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110715/e6c8cec4/attachment.html> From fuzzyman at gmail.com Fri Jul 15 13:33:31 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Fri, 15 Jul 2011 12:33:31 +0100 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> Message-ID: <CAKCKLWz7=ATJWQ5Z=4FJQkwCFpw6NTkDe7E9MpHLTUn=xgKPgQ@mail.gmail.com> On 15 July 2011 12:27, Richard Jones <richard at python.org> wrote: > On 15 July 2011 21:07, Michael Foord <michael at voidspace.org.uk> wrote: > >> The particular error page is just the standard error 500 page served by >> Apache - and it is possible to configure Apache to serve a custom one for >> the pypi subdomain. Not that I know *specifically how* to do that, but I >> know it is possible. :-) >> > > Yeah, that's pretty much where I'm at :-) > > Really, the hard part (because I have no control over it) is setting up the > alias. > > You could ask on pydotorg-www, there are a few admins on there. > > >> A pypi admin address, that just forwards to this list (?) sounds like a >> very good idea (as well). >> > > I'm not sure this list really wants to be caught up in the administrivia of > running pypi :-) > > Isn't it already? Another need we have at webmaster at python.org is somewhere to redirect reports of pypi problems. We often get direct email reports of issues. An email alias I could just send forward those emails to would be very handy. Currently I'm using this list (or telling people to report stuff on the tracker - which is a bit unfriendly for someone who has already taken the time to report an issue). All the best, Michael > > Richard > > >> >> > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110715/fdcb3d31/attachment.html> From mal at egenix.com Fri Jul 15 13:45:22 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Jul 2011 13:45:22 +0200 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> Message-ID: <4E202852.9010609@egenix.com> Richard Jones wrote: > On 15 July 2011 21:07, Michael Foord <michael at voidspace.org.uk> wrote: > >> The particular error page is just the standard error 500 page served by >> Apache - and it is possible to configure Apache to serve a custom one for >> the pypi subdomain. Not that I know *specifically how* to do that, but I >> know it is possible. :-) >> > > Yeah, that's pretty much where I'm at :-) > > Really, the hard part (because I have no control over it) is setting up the > alias. You just need to contact postmaster at python.org. They can set it up for you. >> A pypi admin address, that just forwards to this list (?) sounds like a >> very good idea (as well). >> > > I'm not sure this list really wants to be caught up in the administrivia of > running pypi :-) > > > Richard > > >> >> > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From faassen at startifact.com Fri Jul 15 14:04:34 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 14:04:34 +0200 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <CAHrZfZB+DHR1LQmFmsgCyD0_b21hyQRdmDsG8YFiKTrcqwqMDw@mail.gmail.com> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> <4E135C13.3010207@v.loewis.de> <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> <CAHrZfZB+DHR1LQmFmsgCyD0_b21hyQRdmDsG8YFiKTrcqwqMDw@mail.gmail.com> Message-ID: <ivpac6$kv3$2@dough.gmane.org> On 07/05/2011 11:55 PM, Richard Jones wrote: > Could we rewind right back to the start of this conversation when it > was suggested that you keep and ship a local mirror of packages you > rely on? Such mirroring is trivial to set up. Why force everybody who depends on PyPI making a local mirror? It may be trivial, but trivial * 1000 is still a lot of work. So I'd like to share the burden of setting it up and maintaining it with others. Regards, Martijn From faassen at startifact.com Fri Jul 15 14:01:18 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 14:01:18 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110706011938.0B6C13A4108@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.g mail.com> <20110706011938.0B6C13A4108@sparrow.telecommunity.com> Message-ID: <ivpa62$kv3$1@dough.gmane.org> On 07/06/2011 03:19 AM, P.J. Eby wrote: > At 08:56 PM 7/5/2011 +0200, Martijn Faassen wrote: >> Sure. Unacceptable interference with the freedom of package authors to >> remove stuff. Unacceptable because developers need >> the freedom to remove old versions of their software from the >> internet. :) > > You still seem to be ignoring the part where insisting that people keep > old things around for your convenience *also* means they get support > requests from everyone *else* on the internet. In other words, a package > author has to pay an (ongoing and perpetual) support cost for your > temporary convenience. > > You are not making a principled moral stand here. Rather, you are asking > that other people be made to carry an additional cost, for your benefit, > on the basis that you don't think their cost is important, but you think > your benefit is. A package once released to the Internet can be kept by someone else on the Internet, no matter what the original package author has to say, if that package has a license which permits it. I'm looking for a way to make packages stay unchanged in a way convenient for myself and others. Regards, Martijn P.S. Concerning morality, way off topic. Perhaps reflect on impugning someone's morality as an aspect of moral behavior? Thanks. From benji at benjiyork.com Fri Jul 15 14:12:03 2011 From: benji at benjiyork.com (Benji York) Date: Fri, 15 Jul 2011 08:12:03 -0400 Subject: [Catalog-sig] disallowing the removal of packages? In-Reply-To: <ivpac6$kv3$2@dough.gmane.org> References: <iusmso$6nl$1@dough.gmane.org> <4E121AAB.5010305@v.loewis.de> <CAGT4ZFiV0GqVe2aB3ApPhJ2ESY3PE2ZLTp0EsQgQSdB2WY4PWw@mail.gmail.com> <4E12C299.6000807@v.loewis.de> <CAGT4ZFgfCzz14s08dVUiBjird7s0PhU=+1-XtAuLJ1JGVjr1Dg@mail.gmail.com> <4E134DE2.2090704@v.loewis.de> <CAGT4ZFgW6=hGrfYq6Ye=aosPLC4cTHSJVVDSa=zpu=SsaDX9Cw@mail.gmail.com> <4E13539F.20904@v.loewis.de> <CAGT4ZFhF160urcgY1LBDq8LiCCkiZgE6v8e+R3a+GwOgPgiqmw@mail.gmail.com> <4E135C13.3010207@v.loewis.de> <CAGT4ZFiDS40Uyg5Vo=VfEnsvVXvzYp0mFYLfJcNccUKaymGVpg@mail.gmail.com> <CAHrZfZB+DHR1LQmFmsgCyD0_b21hyQRdmDsG8YFiKTrcqwqMDw@mail.gmail.com> <ivpac6$kv3$2@dough.gmane.org> Message-ID: <CAC8tyQ55+KOOms3HDeZ7eOcZ41hTdZ9KPAm8J5UfXR0+cZAM5w@mail.gmail.com> On Fri, Jul 15, 2011 at 8:04 AM, Martijn Faassen <faassen at startifact.com> wrote: > On 07/05/2011 11:55 PM, Richard Jones wrote: >> >> Could we rewind right back to the start of this conversation when it >> was suggested that you keep and ship a local mirror of packages you >> rely on? Such mirroring is trivial to set up. > > Why force everybody who depends on PyPI making a local mirror? It may be > trivial, but trivial * 1000 is still a lot of work. > > So I'd like to share the burden of setting it up and maintaining it with > others. Indeed. There are enough people that desire a "write only" PyPI that it would be less work globally for those of us that want one to share the burden of building it. On the other hand, even though I'm in the don't-delete-things-from-PyPI camp myself, I'm not sure enough of its users and maintainers share the sentiment that we'll be able to agree to change PyPI to fulfill the roll. We may have to build it outside of PyPI proper. -- Benji York From richard at python.org Fri Jul 15 14:24:53 2011 From: richard at python.org (Richard Jones) Date: Fri, 15 Jul 2011 22:24:53 +1000 Subject: [Catalog-sig] Fwd: Recieved following error at 15:11 EST, 7/8/2011 on pypi.python.org/pypi/ In-Reply-To: <CAKCKLWz7=ATJWQ5Z=4FJQkwCFpw6NTkDe7E9MpHLTUn=xgKPgQ@mail.gmail.com> References: <CAPBNLxkJZDZHCm8FR0S5xnUc-5YkDuhot+FQO9ixKrgMc6m1Cw@mail.gmail.com> <4E175900.6070907@voidspace.org.uk> <CAHrZfZDyV+qWv=Pop0ZrQvBM5OHoTKJxLug+0WAvfN-T6-pYcw@mail.gmail.com> <CAKCKLWyfV7ADVDaca-uJQEe=VboZDFi2w3zGsOv4wOD6=cMw0Q@mail.gmail.com> <CAHrZfZDBWixAz+AoJ3-BiSyr2nueVA2g7e35rpWYT0T0QyE=uQ@mail.gmail.com> <CAKCKLWz7=ATJWQ5Z=4FJQkwCFpw6NTkDe7E9MpHLTUn=xgKPgQ@mail.gmail.com> Message-ID: <CAHrZfZBBnh-aBhT=bpQq4104ZdLhcNK6M4nV+n6rG-2r+AX4pQ@mail.gmail.com> On 15 July 2011 21:33, Michael Foord <fuzzyman at gmail.com> wrote: > > > On 15 July 2011 12:27, Richard Jones <richard at python.org> wrote: >> >> On 15 July 2011 21:07, Michael Foord <michael at voidspace.org.uk> wrote: >>> >>> The particular error page is just the standard error 500 page served by >>> Apache - and it is possible to configure Apache to serve a custom one for >>> the pypi subdomain. Not that I know *specifically how* to do that, but I >>> know it is possible. :-) >> >> Yeah, that's pretty much where I'm at :-) >> Really, the hard part (because I have no control over it) is setting up >> the alias. > > You could ask on pydotorg-www, there are a few admins on there. Indeed that was my next step, but I've been too busy playing Just Cause 2 and trying to download Kirbal Space Program to get around to it :-) Richard From jim at zope.com Fri Jul 15 15:24:54 2011 From: jim at zope.com (Jim Fulton) Date: Fri, 15 Jul 2011 09:24:54 -0400 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) Message-ID: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> On Fri, Jul 15, 2011 at 8:12 AM, Benji York <benji at benjiyork.com> wrote: > On Fri, Jul 15, 2011 at 8:04 AM, Martijn Faassen <faassen at startifact.com> wrote: >> On 07/05/2011 11:55 PM, Richard Jones wrote: >>> >>> Could we rewind right back to the start of this conversation when it >>> was suggested that you keep and ship a local mirror of packages you >>> rely on? Such mirroring is trivial to set up. >> >> Why force everybody who depends on PyPI making a local mirror? It may be >> trivial, but trivial * 1000 is still a lot of work. >> >> So I'd like to share the burden of setting it up and maintaining it with >> others. > > Indeed. ?There are enough people that desire a "write only" PyPI that > it would be less work globally for those of us that want one to share > the burden of building it. > > On the other hand, even though I'm in the > don't-delete-things-from-PyPI camp myself, I'm not sure enough of its > users and maintainers share the sentiment that we'll be able to agree > to change PyPI to fulfill the roll. ?We may have to build it outside > of PyPI proper. You may be right, however: 1. I think if package maintainers actually knew that removing old revisions did harm, most would not delete revisions or packages unless they thought it was necessary. I think most people who remove old revisions do so for cleanliness, not appreciating that there is a downside. 2. I think a lot of people were turned off by the mandatory nature of the original proposal. (I now regret my +1 of that proposal, made in support of the sentiment behind it, but not taking into consideration how it runs counter to Python culture.) PROPOSAL ========= I propose we *try* simply adding an informative confirmation when removing a package or revision that simply explains that removal may inconvenience people relying on the package or the specific version. I predict that there will be a lot fewer annoying removals of we do this. I'd also like to remind everyone that on rare occasions, there are good reasons to remove a package or revision and the sort of mirror contemplated here would make that a lot harder. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From benji at benjiyork.com Fri Jul 15 15:43:47 2011 From: benji at benjiyork.com (Benji York) Date: Fri, 15 Jul 2011 09:43:47 -0400 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> Message-ID: <CAC8tyQ6oBXmGr7OBbAvk_+oqSoZStKQjN7B8uUDa1nk5Tte0Qg@mail.gmail.com> On Fri, Jul 15, 2011 at 9:24 AM, Jim Fulton <jim at zope.com> wrote: > 1. I think if package maintainers actually knew that removing old > revisions did harm, most would not delete revisions or packages unless > they thought it was necessary. ?I think most people who remove old > revisions do so for cleanliness, not appreciating that there is a > downside. > PROPOSAL > ========= > > I propose we *try* simply adding an informative confirmation when > removing a package or revision that simply explains that removal may > inconvenience people relying on the package or the specific version. > I predict that there will be a lot fewer annoying removals of we do > this. +1 This is cheap to do and doesn't restrict a package owner's ability to remove a package if they really feel the need. It'd be really nice if we could quantify the number of packages removed before and after the change, but I'm not sure we collect enough data to do so. -- Benji York From faassen at startifact.com Fri Jul 15 16:36:28 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 16:36:28 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E14851C.6070402@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> Message-ID: <ivpj90$f4e$1@dough.gmane.org> Hi there, On 07/06/2011 05:54 PM, M.-A. Lemburg wrote: >> This is not a good reason, IMHO. You can go on with new versions and a >> new name, maybe you could want to deprecate the old package, but it's >> not a good reason to remove it. > > I think undoing mistakes in package names is a very > good reason to remove them. As package author you don't want such > mistakes to stay on the net forever, if you can avoid it. I don't understand the "you don't want such mistakes to stay on the net forever" line of argumentation. As a package author, once you release something to the internet, you can assume it'll stay on the net. This is already true. So this is an unfixable problem that we're not making worse by having an immutable mirror. >>> * legal action (copyright, trademark, DMCA, license issues, etc) >>> >>> * removal of malicious packages (e.g. script kiddy stuff in >>> setup.py) >>> * seriously broken builds (e.g. that cause users to lose data) >> >> This is would make a good reason for package removal, but not for >> version reassignment. I.E. if I delete version 0.4.3 because it >> deleted my /usr instead of /usr/share/mylib/content, then I would be >> right at removing it, but there's no point in allowing any other >> package to take back 0.4.3. It's simply gone. > > I'm not sure I understand what you want to say. I wasn't talking > about version reassignment in the above cases. Well, it restricts the use cases. The use case "re-release Foo 3.0 but with different contents" doesn't seem to be necessary to tackle the above, just plain removal. So a friendly perpetual mirror would need to be able to handle delete requests but not re-upload requests. >>> * reassigning package names (not sure whether that's possible with >>> PyPI, but it certainly happens in the wild every now and then) >> >> I'm not sure about what you mean here. > > Author A releases a package X, then drops the idea and removes > the package, freeing up the name for others to use. Later on, > author B uses the name X for something different and creates > a new package X with a new set of releases. I wonder by the way whether PyPI supports the "dropping package name forever" use case now. I think a lot is to be gained if you assume package names to be unique forever. For instance: allowing this would be a major security risk: I release package X. Package X is used by people. Then I drop the name. Someone else completely unrelated comes along, creates package with the same name, and uploads evil code. Hilarity ensues. I know the answer already: "Just don't use packages by people who will drop the name at a nebulous point in the future then!" :) Yes, allowing this follows from the "developers should have total freedom" goal that is apparently the main driver behind PyPI's use cases, but there are also "security please" and "repeatability" use cases. Regards, Martijn From faassen at startifact.com Fri Jul 15 16:55:47 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 16:55:47 +0200 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> Message-ID: <ivpkd6$mv1$1@dough.gmane.org> On 07/15/2011 03:24 PM, Jim Fulton wrote: [snip] > 1. I think if package maintainers actually knew that removing old > revisions did harm, most would not delete revisions or packages unless > they thought it was necessary. I think most people who remove old > revisions do so for cleanliness, not appreciating that there is a > downside. > > 2. I think a lot of people were turned off by the mandatory nature of > the original proposal. (I now regret my +1 of that proposal, made in > support of the sentiment behind it, but not taking into consideration > how it runs counter to Python culture.) [snip proposal] Sure, I already indicated a +1 earlier. I tried to sketch out the use cases behind my proposal but people focused on what turned them off. So thinking about this more, what about an extra feature that focuses on communication? What I'm interested in is the following: * reducing the incidence of suddenly removed packages (the message might help) * being aware that packages are removed Others also expressed an interest in having the following: * having a way to deprecate old releases or whole packages So what if we provided communication channels for that? if a package or release is removed by an author, they could be asked to give a reason ("botched upload!", "legal reasons"), whatever. Then there'll be a feed which shows recent deletions along with the reasons. There'd be some machine readable components in this feed too. Also useful for tools would be way to access the total list of all deletions. This could also be expanded to a "deprecation message". Instead of removing a package, an author can add a deprecation message to a package. This could also be on a feed. This way we have a communication channel where we can be informed about what's going on. In response both humans as well as automated tools could take action. You could for instance have a scanning tool which looks at installed packages and tells you which ones are deprecated or removed. It doesn't offer the same guarantees as a perpetual mirror would, so it doesn't quite solve the same use cases. But it would make it possible for people to properly collaborate and communicate about this kind of thing. I'm not proposing that any of the existing client-side tools are adjusted to support any of this. tldr: we've got a feed and index for packages. what about a feed and index for removals and deprecations? Regards, Martijn From g.brandl at gmx.net Fri Jul 15 17:14:09 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 15 Jul 2011 17:14:09 +0200 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> Message-ID: <ivpler$rc5$1@dough.gmane.org> Am 15.07.2011 15:24, schrieb Jim Fulton: > PROPOSAL > ========= > > I propose we *try* simply adding an informative confirmation when > removing a package or revision that simply explains that removal may > inconvenience people relying on the package or the specific version. > I predict that there will be a lot fewer annoying removals of we do > this. That's a very good compromise. +1. Georg From doug.hellmann at gmail.com Fri Jul 15 17:16:57 2011 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Fri, 15 Jul 2011 11:16:57 -0400 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <ivpler$rc5$1@dough.gmane.org> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> <ivpler$rc5$1@dough.gmane.org> Message-ID: <BB8097D4-04AB-4C5C-B6D8-9505DF315B0F@gmail.com> On Jul 15, 2011, at 11:14 AM, Georg Brandl wrote: > Am 15.07.2011 15:24, schrieb Jim Fulton: > >> PROPOSAL >> ========= >> >> I propose we *try* simply adding an informative confirmation when >> removing a package or revision that simply explains that removal may >> inconvenience people relying on the package or the specific version. >> I predict that there will be a lot fewer annoying removals of we do >> this. > > That's a very good compromise. +1. +1 Doug From mal at egenix.com Fri Jul 15 17:59:31 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Jul 2011 17:59:31 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivpj90$f4e$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> Message-ID: <4E2063E3.5080705@egenix.com> Martijn Faassen wrote: > Hi there, > > On 07/06/2011 05:54 PM, M.-A. Lemburg wrote: >>> This is not a good reason, IMHO. You can go on with new versions and a >>> new name, maybe you could want to deprecate the old package, but it's >>> not a good reason to remove it. >> >> I think undoing mistakes in package names is a very >> good reason to remove them. As package author you don't want such >> mistakes to stay on the net forever, if you can avoid it. > > I don't understand the "you don't want such mistakes to stay on the net > forever" line of argumentation. As a package author, once you release > something to the internet, you can assume it'll stay on the net. This is > already true. So this is an unfixable problem that we're not making > worse by having an immutable mirror. It's not unfixable, but it's not easy fo fix either. In any case, we shouldn't be making it harder for authors. We shouldn't forget that we have Python users that are young, not very good at naming things, not aware of consequences that their naming scheme may have, careless, etc. etc. We don't want to make things hard for them, otherwise they'd simply turn away from us, instead of learning how to work with PyPI properly. We do have a responsibility here as community and simply saying: "you uploaded it, so it's your fault, that it'll stay up there", blaming the author forever, does not align well with our Python community spirit. >>>> * legal action (copyright, trademark, DMCA, license issues, etc) >>>> >>>> * removal of malicious packages (e.g. script kiddy stuff in >>>> setup.py) >>>> * seriously broken builds (e.g. that cause users to lose data) >>> >>> This is would make a good reason for package removal, but not for >>> version reassignment. I.E. if I delete version 0.4.3 because it >>> deleted my /usr instead of /usr/share/mylib/content, then I would be >>> right at removing it, but there's no point in allowing any other >>> package to take back 0.4.3. It's simply gone. >> >> I'm not sure I understand what you want to say. I wasn't talking >> about version reassignment in the above cases. > > Well, it restricts the use cases. The use case "re-release Foo 3.0 but > with different contents" doesn't seem to be necessary to tackle the > above, just plain removal. > > So a friendly perpetual mirror would need to be able to handle delete > requests but not re-upload requests. It all depends ... again, it's best to keep the files around that you've tested, rather than relying on some stored on some external file server. >>>> * reassigning package names (not sure whether that's possible with >>>> PyPI, but it certainly happens in the wild every now and then) >>> >>> I'm not sure about what you mean here. >> >> Author A releases a package X, then drops the idea and removes >> the package, freeing up the name for others to use. Later on, >> author B uses the name X for something different and creates >> a new package X with a new set of releases. > > I wonder by the way whether PyPI supports the "dropping package name > forever" use case now. Sure: You file a ticket and Martin or Richard removes the package. > I think a lot is to be gained if you assume package names to be unique > forever. > > For instance: allowing this would be a major security risk: I release > package X. Package X is used by people. Then I drop the name. Someone > else completely unrelated comes along, creates package with the same > name, and uploads evil code. Hilarity ensues. > > I know the answer already: "Just don't use packages by people who will > drop the name at a nebulous point in the future then!" :) As PyPI grows, we'll sooner or later see popular names being reserved on PyPI or names being used for PyPI/distutils/setuptools experiments/learning. Removal of the latter already does occur now, e.g. with the linked list packages described in a Python book for learning distutils packaging. > Yes, allowing this follows from the "developers should have total > freedom" goal that is apparently the main driver behind PyPI's use > cases, but there are also "security please" and "repeatability" use cases. There's no security on PyPI and repeatability is a myth as well :-) Just because we don't have malicious packages on PyPI, doesn't mean it's going to stay like this forever and for repeatability you're better off relying on files you've already downloaded and tested, since it is possible to reupload a package release file with different content, or to change the meta data of a release after its initial upload. I'd suggest you build a read-only PyPI mirror tool that people can download and then use on their private net as they see fit. This solves you use case and that of the other proponents of the read-only PyPI idea, while leaving the mainland PyPI index unchanged. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Fri Jul 15 18:00:05 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Jul 2011 18:00:05 +0200 Subject: [Catalog-sig] (counter-)Proposal: Discouraging removal of packagers and revisions (Was Re: disallowing the removal of packages?) In-Reply-To: <BB8097D4-04AB-4C5C-B6D8-9505DF315B0F@gmail.com> References: <CAPDm-FgEbi838bq7=+gGChXnrpJT-4c+TpcqZOiVcMu6=ON9ng@mail.gmail.com> <ivpler$rc5$1@dough.gmane.org> <BB8097D4-04AB-4C5C-B6D8-9505DF315B0F@gmail.com> Message-ID: <4E206405.6070103@egenix.com> Doug Hellmann wrote: > > On Jul 15, 2011, at 11:14 AM, Georg Brandl wrote: > >> Am 15.07.2011 15:24, schrieb Jim Fulton: >> >>> PROPOSAL >>> ========= >>> >>> I propose we *try* simply adding an informative confirmation when >>> removing a package or revision that simply explains that removal may >>> inconvenience people relying on the package or the specific version. >>> I predict that there will be a lot fewer annoying removals of we do >>> this. >> >> That's a very good compromise. +1. > > +1 +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From faassen at startifact.com Fri Jul 15 18:20:53 2011 From: faassen at startifact.com (Martijn Faassen) Date: Fri, 15 Jul 2011 18:20:53 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E2063E3.5080705@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> Message-ID: <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> Hi there, On Fri, Jul 15, 2011 at 5:59 PM, M.-A. Lemburg <mal at egenix.com> wrote: >> Yes, allowing this follows from the "developers should have total >> freedom" goal that is apparently the main driver behind PyPI's use >> cases, but there are also "security please" and "repeatability" use cases. > > There's no security on PyPI and repeatability is a myth as well :-) Yes, there is no security and repeatability *now*. That's why I started this whole discussion in the first place, right? You need to argue why these are *not use cases* for PyPI, not that they're not there now. Because it seems if PyPI doesn't care about those, PyPI is really not usable for automated downloading of packages at all. And it seems that is a use case of PyPI? > Just because we don't have malicious packages on PyPI, doesn't > mean it's going to stay like this forever and for repeatability > you're better off relying on files you've already downloaded and > tested, since it is possible to reupload a package release file > with different content, or to change the meta data of a release > after its initial upload. Yes, I talked about reuploading packages and the risks involved in that before, but at least that's not possible by someone else who has nothing to do with the removed package in the first place. Removing a package entirely and letting the name be reused opens up the possibility for completely unrelated people to come in and do nasty things. > I'd suggest you build a read-only PyPI mirror tool that > people can download and then use on their private net as > they see fit. This solves you use case and that of the other > proponents of the read-only PyPI idea, while leaving the > mainland PyPI index unchanged. I will explain again why this doesn't solve my use case. I don't work in a vacuum. I share code with others. This code has dependencies on other code. So how do people obtain this other code? With a private mirror, I'd need to first run all that infrastructure, and then give people access to my private mirror. How private is it going to stay then? How many people are using my code? What maintenance burden does that entail for me? Or I need to give them a possibly giant tarball of all the packages I'm using. Is that really the only way forward for sharing code? Even for libraries with dependencies? PyPI I thought was among other things central place where people can download and install packages from so that they can resolve dependencies, but you seem to be arguing against doing that. At most it's some kind of showcase for packages that peoples should take into their consideration. Taking this point to the extreme, it's *never* something that you can automate downloading from. Instead you should be giving a giant tarball of packages to everybody, always, if they use your code at all. Then this will need to be explained to the people who *are* using PyPI for this purpose. I.e. some significant fraction of all Python programmers out there. Regards, Martijn From martin at v.loewis.de Fri Jul 15 22:58:31 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 15 Jul 2011 22:58:31 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivpj90$f4e$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> Message-ID: <4E20A9F7.1050000@v.loewis.de> >> Author A releases a package X, then drops the idea and removes >> the package, freeing up the name for others to use. Later on, >> author B uses the name X for something different and creates >> a new package X with a new set of releases. > > I wonder by the way whether PyPI supports the "dropping package name > forever" use case now. Of course. When you delete a package, all traces of it are deleted from PyPI for good, except for a log record stating that the package was deleted (and by whom, which again isn't published). Regards, Martin From ben+python at benfinney.id.au Sat Jul 16 01:08:31 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 16 Jul 2011 09:08:31 +1000 Subject: [Catalog-sig] an immutable mirror of PyPI References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> Message-ID: <87aacfb0r4.fsf@benfinney.id.au> Martijn Faassen <faassen at startifact.com> writes: > I don't work in a vacuum. I share code with others. This code has > dependencies on other code. So how do people obtain this other code? By depending on other code, you have a choice to make: you either take the maintenance burden on yourself, or you delegate the maintenance burden (usually to the developers of that code). By delegating the maintenance burden of that code elsewhere, that entails delegating the responsibility for future availability of that code. > PyPI I thought was among other things central place where people can > download and install packages from so that they can resolve > dependencies, but you seem to be arguing against doing that. I find it strange that I'm defending PyPI in this instance, since I am quite sympathetic to complaints that it has poor policies on package availability and many other complaints. But you seem to expect that PyPI must guarantee that any package version ever available will be available forever. That's not reasonable, I think. Instead, you need to choose packages considering whether you trust the package to remain available, which is a social issue between you and the people developing that work. If you think there is a significant risk the people responsible for that package will remove a version on which you depend from PyPI, you should engage in dialogue with those people to resolve that. I don't think PyPI has any business requiring package developers to keep a version available at PyPI beyond when they want it available there. The risks inherent in that need to be addressed as a social issue, not a technical limitation. > At most it's some kind of showcase for packages that peoples should > take into their consideration. Taking this point to the extreme, it's > *never* something that you can automate downloading from. There are points that can be made toward that view; but I don't find this specific case (wanting guaranteed availability of every version forever at PyPI) supports it. > Instead you should be giving a giant tarball of packages to everybody, > always, if they use your code at all. This is indeed a terrible option, and I lament it whenever I see it. I prefer supporting the efforts of those who *do* provide reasonable guarantees of package selection and availability and integration testing. We call them ?operating system distributions?. -- \ ?In general my children refuse to eat anything that hasn't | `\ danced on television.? ?Erma Bombeck | _o__) | Ben Finney From faassen at startifact.com Sat Jul 16 12:54:53 2011 From: faassen at startifact.com (Martijn Faassen) Date: Sat, 16 Jul 2011 12:54:53 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <87aacfb0r4.fsf@benfinney.id.au> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> <87aacfb0r4.fsf@benfinney.id.au> Message-ID: <ivrql8$oq2$1@dough.gmane.org> On 07/16/2011 01:08 AM, Ben Finney wrote: > Martijn Faassen<faassen at startifact.com> writes: > >> I don't work in a vacuum. I share code with others. This code has >> dependencies on other code. So how do people obtain this other code? > > By depending on other code, you have a choice to make: you either take > the maintenance burden on yourself, or you delegate the maintenance > burden (usually to the developers of that code). > > By delegating the maintenance burden of that code elsewhere, that > entails delegating the responsibility for future availability of that > code. There is maintenance burden and there is the package actually existing for download. When I depend on Foo 1.1, I am not delegating maintenance burden to the original developer, unless I go and ask questions about Foo 1.1. The answer can then be: Foo 1.1 is not maintained, sorry. Only when I am interested in upgrades does the original developer come in again. I don't see why these two should be the same: the future availability of an existing release of a package is not identical to continued development of that code. >> PyPI I thought was among other things central place where people can >> download and install packages from so that they can resolve >> dependencies, but you seem to be arguing against doing that. > > I find it strange that I'm defending PyPI in this instance, since I am > quite sympathetic to complaints that it has poor policies on package > availability and many other complaints. > > But you seem to expect that PyPI must guarantee that any package version > ever available will be available forever. That's not reasonable, I > think. I am not barging in here with expectations. I'm coming in here with use cases and proposals. It seems my use cases are rejected as goals of PyPI. In that case I want to get a better understanding of the goals of PyPI. You say that the goal of perpetual availability of packages is an unreasonable goal of PyPI or related services. You don't seem to explain why. So I have use cases: I can release code that relies on releases that can disappear or can be replaced. I think this is bad for repeatability and security. I'd like to see some improvements made. How would we make these improvements? I've so far proposed three ideas: * PyPI not throwing away things after a grace period. Almost universally rejected idea * an additional service, a mirror, that offers some repeatability guarantees. Removal would need to go through channels, implying some kind of custodianship I think people here are wary about. * better communication channels: a list of what's been removed, a list of what's been deprecated. I can then write tools that help me maintain my projects. It's not the same as the above ideas: old projects can still break at the whim of people whose code I depend on, but it'll at least help manage this issue. But perhaps you have better ideas on how to better help manage this. I am getting a bit tired of hearing "you can do this yourself", as this ties into to heart of collaboration, and PyPI if anything is at least supports collaboration. > Instead, you need to choose packages considering whether you trust the > package to remain available, which is a social issue between you and the > people developing that work. > If you think there is a significant risk the people responsible for that > package will remove a version on which you depend from PyPI, you should > engage in dialogue with those people to resolve that. And how exactly am I supposed to read people's minds, possibly years into the future? I had absolutely no expectation that this would happen with the release that disappeared on over a month ago. The developer one day just decided to clean up old, unsupported releases. Of course I contacted the developer after it happened. Several others did too. I then started thinking about how to reduce this risk in a more broad sense. > I don't think PyPI has any business requiring package developers to keep > a version available at PyPI beyond when they want it available there. > The risks inherent in that need to be addressed as a social issue, not a > technical limitation. Yes, this is a social issue. But tools can support social issues. If people tell me to keep my own private mirror, that's a tool solution too, but not a very social one. >> At most it's some kind of showcase for packages that peoples should >> take into their consideration. Taking this point to the extreme, it's >> *never* something that you can automate downloading from. > > There are points that can be made toward that view; but I don't find > this specific case (wanting guaranteed availability of every version > forever at PyPI) supports it. > >> Instead you should be giving a giant tarball of packages to everybody, >> always, if they use your code at all. > > This is indeed a terrible option, and I lament it whenever I see it. > > I prefer supporting the efforts of those who *do* provide reasonable > guarantees of package selection and availability and integration > testing. We call them ?operating system distributions?. The requirements for developers concerning library availability are not identical to those of users. Operating system distributions focus on a stable platform for users. Some developers need to develop cross-platform code. Some developers need to develop different versions of a project, or different project that rely on different versions of dependencies. Some developers need to depend on libraries or library versions not (yet) available in distributions. These developers effectively create a stable distribution of dependencies that they have tested together. It's useful to have tools to support this and allow these developers to share their code with others. Regards, Martijn From faassen at startifact.com Sat Jul 16 12:58:36 2011 From: faassen at startifact.com (Martijn Faassen) Date: Sat, 16 Jul 2011 12:58:36 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E20A9F7.1050000@v.loewis.de> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> Message-ID: <ivrqs7$ppb$1@dough.gmane.org> On 07/15/2011 10:58 PM, "Martin v. L?wis" wrote: >>> Author A releases a package X, then drops the idea and removes >>> the package, freeing up the name for others to use. Later on, >>> author B uses the name X for something different and creates >>> a new package X with a new set of releases. >> >> I wonder by the way whether PyPI supports the "dropping package name >> forever" use case now. > > Of course. When you delete a package, all traces of it are deleted from > PyPI for good, except for a log record stating that the package was > deleted (and by whom, which again isn't published). Okay, so this scenario is possible: * developer of a popular package gets fed up for unknown reasons * removes his package from PyPI (not realizing the thing below) * someone else notices this and recreates the package maliciously * people who download the package (possibly indirectly, by downloading a library that uses this as a dependency) will be bitten It's not an extremely likely scenario but as PyPI grows it becomes possible. I wonder whether there are tooling solutions possible to detect this before it's too late. A public log of what got removed would be useful so people can keep an eye on things - but for this to be caught it would mean that the log would need to include recreations as well. Regards, Martijn From benji at benjiyork.com Sat Jul 16 18:40:16 2011 From: benji at benjiyork.com (Benji York) Date: Sat, 16 Jul 2011 12:40:16 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivrqs7$ppb$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> Message-ID: <CAC8tyQ6Di2Xh4+5EQ_2fLk9MP9rj4J6q8TNfVaB9DMkyLeboVw@mail.gmail.com> On Sat, Jul 16, 2011 at 6:58 AM, Martijn Faassen <faassen at startifact.com> wrote: > I wonder whether there are tooling solutions possible to detect this before > it's too late. A public log of what got removed would be useful so people > can keep an eye on things - but for this to be caught it would mean that the > log would need to include recreations as well. Being a buildout user, if I were to tackle that I'd add something along the lines of SSH's warnings when a host fingerprint changes. I.e., require that package hashes be given (much like you can require that versions be specified) and check those on download. -- Benji York From faassen at startifact.com Sat Jul 16 18:50:07 2011 From: faassen at startifact.com (Martijn Faassen) Date: Sat, 16 Jul 2011 18:50:07 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <CAC8tyQ6Di2Xh4+5EQ_2fLk9MP9rj4J6q8TNfVaB9DMkyLeboVw@mail.gmail.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <CAC8tyQ6Di2Xh4+5EQ_2fLk9MP9rj4J6q8TNfVaB9DMkyLeboVw@mail.gmail.com> Message-ID: <CAGT4ZFgFaXVVr8=e2R81LMZ_YSYtMzcmnroOgvqj1SmmAU7C0A@mail.gmail.com> Hey, On Sat, Jul 16, 2011 at 6:40 PM, Benji York <benji at benjiyork.com> wrote: > On Sat, Jul 16, 2011 at 6:58 AM, Martijn Faassen <faassen at startifact.com> wrote: >> I wonder whether there are tooling solutions possible to detect this before >> it's too late. A public log of what got removed would be useful so people >> can keep an eye on things - but for this to be caught it would mean that the >> log would need to include recreations as well. > > Being a buildout user, if I were to tackle that I'd add something along > the lines of SSH's warnings when a host fingerprint changes. ?I.e., > require that package hashes be given (much like you can require that > versions be specified) and check those on download. Yes, for changes this would be possible (assuming hashes). Removals by themselves are another problem, though. Regards, Martijn From tjreedy at udel.edu Mon Jul 18 22:43:34 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 18 Jul 2011 16:43:34 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivrqs7$ppb$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> Message-ID: <j025tl$d9g$1@dough.gmane.org> On 7/16/2011 6:58 AM, Martijn Faassen wrote: > Okay, so this scenario is possible: > > * developer of a popular package gets fed up for unknown reasons > > * removes his package from PyPI (not realizing the thing below) > > * someone else notices this and recreates the package maliciously pypi could prohibit the reuse of deleted package names. If a name was 'retired' for legal reasons, then it should stay retired anyway. -- Terry Jan Reedy From tjreedy at udel.edu Mon Jul 18 22:47:58 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 18 Jul 2011 16:47:58 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <ivrql8$oq2$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> <87aacfb0r4.fsf@benfinney.id.au> <ivrql8$oq2$1@dough.gmane.org> Message-ID: <j0265t$fmh$1@dough.gmane.org> On 7/16/2011 6:54 AM, Martijn Faassen wrote: > So I have use cases: I can release code that relies on releases that can > disappear or can be replaced. I think this is bad for repeatability and > security. I'd like to see some improvements made. How would we make > these improvements? I've so far proposed three ideas: > > * PyPI not throwing away things after a grace period. Almost universally > rejected idea > > * an additional service, a mirror, that offers some repeatability > guarantees. Removal would need to go through channels, implying some > kind of custodianship I think people here are wary about. > > * better communication channels: a list of what's been removed, a list > of what's been deprecated. I can then write tools that help me maintain > my projects. It's not the same as the above ideas: old projects can > still break at the whim of people whose code I depend on, but it'll at > least help manage this issue. If packages have dependency list, then pypi could keep a ref count for each package and either not allow deletion of packages with a + count or require admin approval or at least inform the developer 'You are about to delete a package that other programs depend on: you sure you want to delete it?' followed 'Are you really really sure? Can we keep the file for a few months and give users a warning?" -- Terry Jan Reedy From mal at egenix.com Tue Jul 19 00:04:09 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Jul 2011 00:04:09 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <j025tl$d9g$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> Message-ID: <4E24ADD9.4080106@egenix.com> Terry Reedy wrote: > On 7/16/2011 6:58 AM, Martijn Faassen wrote: > >> Okay, so this scenario is possible: >> >> * developer of a popular package gets fed up for unknown reasons >> >> * removes his package from PyPI (not realizing the thing below) >> >> * someone else notices this and recreates the package maliciously > > pypi could prohibit the reuse of deleted package names. > If a name was 'retired' for legal reasons, then it should stay retired > anyway. Recycling of package names can very well have a real and honest background, e.g. if someone decides to give a package name to someone else for whatever reason. Happens in DNS all the time. BTW: To address your repeatability/security concerns, the tools you are using would also have to store the hash check sum of the downloaded packages together with the version. AFAIK, buildout only pins down versions, not MD5/SHA1 sums. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From faassen at startifact.com Tue Jul 19 18:11:23 2011 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 19 Jul 2011 18:11:23 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <j0265t$fmh$1@dough.gmane.org> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E2063E3.5080705@egenix.com> <CAGT4ZFjCa=qJ0rKPGEiJ5rvvQcB=nRiqBXgcMZwUJUoE8pyNnw@mail.gmail.com> <87aacfb0r4.fsf@benfinney.id.au> <ivrql8$oq2$1@dough.gmane.org> <j0265t$fmh$1@dough.gmane.org> Message-ID: <j04a9j$qr7$1@dough.gmane.org> On 07/18/2011 10:47 PM, Terry Reedy wrote: > If packages have dependency list, then pypi could keep a ref count for > each package and either not allow deletion of packages with a + count or > require admin approval or at least inform the developer 'You are about > to delete a package that other programs depend on: you sure you want to > delete it?' followed 'Are you really really sure? Can we keep the file > for a few months and give users a warning?" An interesting idea. Wouldn't work for code that isn't on PyPI (an internal project or whatever). It also would work less well with removing older versions: show it forbid throwing away 3.2 if there's a 3.3, or not? Requirements along the line of foo <= 3.2 exist, but are not very common, and cannot anticipate changes in future releases. (giving users a warning is another feature we don't have now in PyPI) Regards, Martijn From chris at simplistix.co.uk Tue Jul 19 19:46:38 2011 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 19 Jul 2011 18:46:38 +0100 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E24ADD9.4080106@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> Message-ID: <4E25C2FE.7050405@simplistix.co.uk> On 18/07/2011 23:04, M.-A. Lemburg wrote: > BTW: To address your repeatability/security concerns, the tools you are > using would also have to store the hash check sum of the downloaded > packages together with the version. AFAIK, buildout only pins down > versions, not MD5/SHA1 sums. I'm pretty sure there's a hashing extension for buildout downloads. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From mal at egenix.com Wed Jul 20 10:54:00 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Jul 2011 10:54:00 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E25C2FE.7050405@simplistix.co.uk> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> <4E25C2FE.7050405@simplistix.co.uk> Message-ID: <4E2697A8.1090505@egenix.com> Chris Withers wrote: > On 18/07/2011 23:04, M.-A. Lemburg wrote: >> BTW: To address your repeatability/security concerns, the tools you are >> using would also have to store the hash check sum of the downloaded >> packages together with the version. AFAIK, buildout only pins down >> versions, not MD5/SHA1 sums. > > I'm pretty sure there's a hashing extension for buildout downloads. You mean: an extension that allow pinning versions and hashes or just one that checks the downloads against the hashes provided by the index server (if it does) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 20 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From chris at simplistix.co.uk Wed Jul 20 10:55:09 2011 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 20 Jul 2011 09:55:09 +0100 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E2697A8.1090505@egenix.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> <4E25C2FE.7050405@simplistix.co.uk> <4E2697A8.1090505@egenix.com> Message-ID: <4E2697ED.8010901@simplistix.co.uk> On 20/07/2011 09:54, M.-A. Lemburg wrote: > You mean: an extension that allow pinning versions buildout allows version pinning out of the box. > just one that checks the downloads against the hashes provided by > the index server (if it does) ? yes, I think, but I'm not 100% cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From pje at telecommunity.com Wed Jul 20 16:09:44 2011 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 20 Jul 2011 10:09:44 -0400 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <4E2697ED.8010901@simplistix.co.uk> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> <4E25C2FE.7050405@simplistix.co.uk> <4E2697A8.1090505@egenix.com> <4E2697ED.8010901@simplistix.co.uk> Message-ID: <20110720141023.070F03A409B@sparrow.telecommunity.com> At 09:55 AM 7/20/2011 +0100, Chris Withers wrote: >On 20/07/2011 09:54, M.-A. Lemburg wrote: >>You mean: an extension that allow pinning versions > >buildout allows version pinning out of the box. > >>just one that checks the downloads against the hashes provided by >>the index server (if it does) ? > >yes, I think, but I'm not 100% Buildout uses setuptools, and setuptools checks the hashes provided by the index server. So, by default, buildout is probably checking them. From mal at egenix.com Wed Jul 20 16:29:23 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Jul 2011 16:29:23 +0200 Subject: [Catalog-sig] an immutable mirror of PyPI In-Reply-To: <20110720141023.070F03A409B@sparrow.telecommunity.com> References: <iut7ss$bmc$1@dough.gmane.org> <4E124337.4020001@egenix.com> <iuuu85$vmr$1@dough.gmane.org> <4E12FFB9.40603@egenix.com> <CAPDm-FiSpHahpnD-4P9Ga3_tD_uuzR7o3W90a49mM=Pbz3zQBw@mail.gmail.com> <20110705144311.B7CD43A404D@sparrow.telecommunity.com> <iuvb3p$js1$1@dough.gmane.org> <4E135148.7020204@v.loewis.de> <CAGT4ZFg_zVs4XPFN9_oGD6unyGtrFCSE3Jr2RSrA5DffkKN2nQ@mail.gmail.com> <4E135B16.5070805@v.loewis.de> <CAGT4ZFhX7tVtbbpnHAJKRZ9rDODN9WgxfPrsh7Ei+R0RawMg7Q@mail.gmail.com> <4E142669.7090307@egenix.com> <CAF3z5=mFjynVV8gDOg+4+uyJHp_NodSQ27G0arVOwB2VmzAdPw@mail.gmail.com> <4E14851C.6070402@egenix.com> <ivpj90$f4e$1@dough.gmane.org> <4E20A9F7.1050000@v.loewis.de> <ivrqs7$ppb$1@dough.gmane.org> <j025tl$d9g$1@dough.gmane.org> <4E24ADD9.4080106@egenix.com> <4E25C2FE.7050405@simplistix.co.uk> <4E2697A8.1090505@egenix.com> <4E2697ED.8010901@simplistix.co.uk> <20110720141023.070F03A409B@sparrow.telecommunity.com> Message-ID: <4E26E643.1060902@egenix.com> > At 09:55 AM 7/20/2011 +0100, Chris Withers wrote: >> On 20/07/2011 09:54, M.-A. Lemburg wrote: >>> You mean: an extension that allow pinning versions >> >> buildout allows version pinning out of the box. Right, but does it also allow pinning down the hashes to make sure that the downloads are indeed the ones you expect ? That would be essential to provide security for buildout-based configurations that pull their data from servers not under control of the buildout user. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 20 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/