From donald.stufft at gmail.com  Wed Feb  1 00:43:56 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 31 Jan 2012 18:43:56 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
Message-ID: <FA61D66685314D12A150635916BB50FD@gmail.com>

I don't think anyone is arguing that it's not occasionally useful. The question to answer is the occasional usefulness worth the risks that come with it. In my opinion the small utility (being able to correct a borked packaging job) is not worth the risks to both my applications stability, and the security of my entire system. 


On Tuesday, January 31, 2012 at 5:53 PM, Michael Foord wrote:

> 
> 
> On 29 January 2012 23:47, Richard Jones <r1chardj0n3s at gmail.com (mailto:r1chardj0n3s at gmail.com)> wrote:
> > Hi catalog-sig,
> > 
> > When we initially implemented file upload to PyPI it was our intention
> > that the file be immutable once uploaded. The goal was to make things
> > significantly simpler for end users - there would only ever be one
> > file with a given name. If the content changed then so must the name
> > (typically by creating a new release version.)
> > 
> > After the upload facility was put in place we also added the ability
> > to delete files uploaded to pypi. This created a loophole: if a
> > package owner knew how to they could delete the file and re-upload,
> > thus circumventing the replacement protection.
> > 
> > I'm considering closing this loophole by retaining a record of the
> > uploaded file (though not the contents) so that future uploads with
> > the same name wouldn't be allowed. I understand that this is how the
> > ruby gem archive handles deletion of files.
> > 
> > Your thoughts?
> 
> 
> FWIW I've occasionally found it useful to be able to delete uploads and replace them, so I'm -1 on losing this capability.
> 
> All the best,
> 
> Michael
>  
> > 
> > 
> >     Richard
> > _______________________________________________
> > Catalog-SIG mailing list
> > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> > http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 
> 
> -- 
> http://www.voidspace.org.uk/
> 
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
> May you share freely, never taking more than you give.
> -- the sqlite blessing http://www.sqlite.org/different.html 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120131/6e942316/attachment.html>

From tjreedy at udel.edu  Wed Feb  1 01:40:08 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 31 Jan 2012 19:40:08 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <FA61D66685314D12A150635916BB50FD@gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
	<FA61D66685314D12A150635916BB50FD@gmail.com>
Message-ID: <jga1lb$66k$1@dough.gmane.org>

On 1/31/2012 6:43 PM, Donald Stufft wrote:
> I don't think anyone is arguing that it's not occasionally useful. The
> question to answer is the occasional usefulness worth the risks that
> come with it. In my opinion the small utility (being able to correct a
> borked packaging job) is not worth the risks to both my applications
> stability, and the security of my entire system.

The question is whether, on each issue, PyPI should be optimized for 
authors (who provide their modules for free) or for users. Both choices 
are defensible. However, if all choices are made in favor of users, 
there will very likely be fewer things uploaded or even listed, which is 
not favorable for users.

It is hard to take your security concerns too seriously when you 
consistently ignore security suggestions. Prohibiting deletion or 
replacement by authors will give you no protection against the site 
being compromised by other means, whereas the suggestions you ignore would.

-- 
Terry Jan Reedy


From donald.stufft at gmail.com  Wed Feb  1 01:41:32 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 31 Jan 2012 19:41:32 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <jga1lb$66k$1@dough.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
	<FA61D66685314D12A150635916BB50FD@gmail.com>
	<jga1lb$66k$1@dough.gmane.org>
Message-ID: <11F07C0AA2CC467384F7C5BEBBACC120@gmail.com>

Which suggestions did I ignore? 


On Tuesday, January 31, 2012 at 7:40 PM, Terry Reedy wrote:

> On 1/31/2012 6:43 PM, Donald Stufft wrote:
> > I don't think anyone is arguing that it's not occasionally useful. The
> > question to answer is the occasional usefulness worth the risks that
> > come with it. In my opinion the small utility (being able to correct a
> > borked packaging job) is not worth the risks to both my applications
> > stability, and the security of my entire system.
> > 
> 
> 
> The question is whether, on each issue, PyPI should be optimized for 
> authors (who provide their modules for free) or for users. Both choices 
> are defensible. However, if all choices are made in favor of users, 
> there will very likely be fewer things uploaded or even listed, which is 
> not favorable for users.
> 
> It is hard to take your security concerns too seriously when you 
> consistently ignore security suggestions. Prohibiting deletion or 
> replacement by authors will give you no protection against the site 
> being compromised by other means, whereas the suggestions you ignore would.
> 
> -- 
> Terry Jan Reedy
> 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120131/46a5fec5/attachment-0001.html>

From donald.stufft at gmail.com  Wed Feb  1 01:43:49 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 31 Jan 2012 19:43:49 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <jga1lb$66k$1@dough.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
	<FA61D66685314D12A150635916BB50FD@gmail.com>
	<jga1lb$66k$1@dough.gmane.org>
Message-ID: <2F75E8A112BF4A87A824FE05452D06F3@gmail.com>

and by all means, a lot of things aren't protected when the server itself is compromised, we should go ahead and disable any of those that are even mildly annoying too. 


On Tuesday, January 31, 2012 at 7:40 PM, Terry Reedy wrote:

> On 1/31/2012 6:43 PM, Donald Stufft wrote:
> > I don't think anyone is arguing that it's not occasionally useful. The
> > question to answer is the occasional usefulness worth the risks that
> > come with it. In my opinion the small utility (being able to correct a
> > borked packaging job) is not worth the risks to both my applications
> > stability, and the security of my entire system.
> > 
> 
> 
> The question is whether, on each issue, PyPI should be optimized for 
> authors (who provide their modules for free) or for users. Both choices 
> are defensible. However, if all choices are made in favor of users, 
> there will very likely be fewer things uploaded or even listed, which is 
> not favorable for users.
> 
> It is hard to take your security concerns too seriously when you 
> consistently ignore security suggestions. Prohibiting deletion or 
> replacement by authors will give you no protection against the site 
> being compromised by other means, whereas the suggestions you ignore would.
> 
> -- 
> Terry Jan Reedy
> 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120131/d0e95611/attachment.html>

From tjreedy at udel.edu  Wed Feb  1 01:46:53 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 31 Jan 2012 19:46:53 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <11F07C0AA2CC467384F7C5BEBBACC120@gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
	<FA61D66685314D12A150635916BB50FD@gmail.com>
	<jga1lb$66k$1@dough.gmane.org>
	<11F07C0AA2CC467384F7C5BEBBACC120@gmail.com>
Message-ID: <jga220$8j8$1@dough.gmane.org>

1. Record and check md5 hash on all downloads.
2. Redistribute files yourself (if license allows).

Ignore in sense of not respond why not adequate alternative to your request.

It is confusing.
Please do not top post

On 1/31/2012 7:41 PM, Donald Stufft wrote:
> Which suggestions did I ignore?
>
> On Tuesday, January 31, 2012 at 7:40 PM, Terry Reedy wrote:
>> It is hard to take your security concerns too seriously when you
>> consistently ignore security suggestions. Prohibiting deletion or
>> replacement by authors will give you no protection against the site
>> being compromised by other means, whereas the suggestions you ignore
>> would.
-- 
Terry Jan Reedy


From donald.stufft at gmail.com  Wed Feb  1 01:58:14 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 31 Jan 2012 19:58:14 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <jga220$8j8$1@dough.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
	<FA61D66685314D12A150635916BB50FD@gmail.com>
	<jga1lb$66k$1@dough.gmane.org>
	<11F07C0AA2CC467384F7C5BEBBACC120@gmail.com>
	<jga220$8j8$1@dough.gmane.org>
Message-ID: <18E37B9D006C46C39ECA169FECA785A8@gmail.com>

On Tuesday, January 31, 2012 at 7:46 PM, Terry Reedy wrote:
> 1. Record and check md5 hash on all downloads.
> 2. Redistribute files yourself (if license allows).
> 
> Ignore in sense of not respond why not adequate alternative to your request.
> 
> It is confusing.
> Please do not top post
> 
> On 1/31/2012 7:41 PM, Donald Stufft wrote:
> > Which suggestions did I ignore?
> > 
> > On Tuesday, January 31, 2012 at 7:40 PM, Terry Reedy wrote:
> > > It is hard to take your security concerns too seriously when you
> > > consistently ignore security suggestions. Prohibiting deletion or
> > > replacement by authors will give you no protection against the site
> > > being compromised by other means, whereas the suggestions you ignore
> > > would.
> > > 
> > 
> > 
> 
> -- 
> Terry Jan Reedy
> 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 

Email client defaults to top posting. 

1. Pip doesn't support this that i'm aware of. I'm looking at the possibility of adding that to pip but currently I believe it would require zc.buildout.
2. I already do this. This is currently the best option available to people but it is a poor option. It essentially equates too "Well Yes PyPI is insecure by design, if you want security don't use it."

I'm also not arguing for just myself. I use the term "me" and "my" but they are placeholders for "anyone using this system". Unless you think that anyone wanting to not be vulnerable to their app breaking without warning, and without anything changing on their end (besides a new install) and wanting to not be vulnerable to the security issues should just "not use PyPI" which is completely unreasonable.

The *best* place to fix this is in PyPI. That way the fix to these vulnerabilities will be applied for *everyone*. Yes I can work around it on a personal level, but that doesn't help the community, it only helps myself.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120131/5aadc42d/attachment.html>

From jim at zope.com  Wed Feb  1 02:14:48 2012
From: jim at zope.com (Jim Fulton)
Date: Tue, 31 Jan 2012 20:14:48 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
Message-ID: <CAPDm-FgrOd7EFdP60XKieOds4nhyAaX7h_7Nbc-iYrzLHF+egA@mail.gmail.com>

On Sun, Jan 29, 2012 at 6:47 PM, Richard Jones <r1chardj0n3s at gmail.com> wrote:
...
> Your thoughts?

I appreciate Donald's persistence. :)

I'm still +1

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From aclark at aclark.net  Wed Feb  1 02:45:18 2012
From: aclark at aclark.net (Alex Clark)
Date: Tue, 31 Jan 2012 20:45:18 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
Message-ID: <jga5ff$so0$1@dough.gmane.org>

On 1/29/12 6:47 PM, Richard Jones wrote:
> Hi catalog-sig,
>
> When we initially implemented file upload to PyPI it was our intention
> that the file be immutable once uploaded. The goal was to make things
> significantly simpler for end users - there would only ever be one
> file with a given name. If the content changed then so must the name
> (typically by creating a new release version.)
>
> After the upload facility was put in place we also added the ability
> to delete files uploaded to pypi. This created a loophole: if a
> package owner knew how to they could delete the file and re-upload,
> thus circumventing the replacement protection.
>
> I'm considering closing this loophole by retaining a record of the
> uploaded file (though not the contents) so that future uploads with
> the same name wouldn't be allowed. I understand that this is how the
> ruby gem archive handles deletion of files.
>
> Your thoughts?

A belated +1. Given that it's a known "best practice" to bump versions 
whenever you fix a brown bag release, I don't see any valid reason not 
to enforce this.


Alex


>
>
>       Richard


-- 
Alex Clark ? http://pythonpackages.com


From a.badger at gmail.com  Wed Feb  1 03:08:42 2012
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Tue, 31 Jan 2012 18:08:42 -0800
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <jga5ff$so0$1@dough.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org>
Message-ID: <20120201020841.GF23782@unaka.lan>

On Tue, Jan 31, 2012 at 08:45:18PM -0500, Alex Clark wrote:
> On 1/29/12 6:47 PM, Richard Jones wrote:
> >Hi catalog-sig,
> >
> >When we initially implemented file upload to PyPI it was our intention
> >that the file be immutable once uploaded. The goal was to make things
> >significantly simpler for end users - there would only ever be one
> >file with a given name. If the content changed then so must the name
> >(typically by creating a new release version.)
> >
> >After the upload facility was put in place we also added the ability
> >to delete files uploaded to pypi. This created a loophole: if a
> >package owner knew how to they could delete the file and re-upload,
> >thus circumventing the replacement protection.
> >
> >I'm considering closing this loophole by retaining a record of the
> >uploaded file (though not the contents) so that future uploads with
> >the same name wouldn't be allowed. I understand that this is how the
> >ruby gem archive handles deletion of files.
> >
> >Your thoughts?
> 
> A belated +1. Given that it's a known "best practice" to bump
> versions whenever you fix a brown bag release, I don't see any valid
> reason not to enforce this.
> 
One problem I've encountered that "requires" re-uploading is forgetting to
sign my sdists when doing python setup.py sdist upload.  There's probably
a way to use the webui to add the signature after the fact but I haven't
found a way to sign the existing sdist and upload that signature from the
command line.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120131/680aca56/attachment.pgp>

From richard at python.org  Wed Feb  1 03:30:11 2012
From: richard at python.org (Richard Jones)
Date: Wed, 1 Feb 2012 13:30:11 +1100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <20120201020841.GF23782@unaka.lan>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
Message-ID: <CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>

On 1 February 2012 13:08, Toshio Kuratomi <a.badger at gmail.com> wrote:
> One problem I've encountered that "requires" re-uploading is forgetting to
> sign my sdists when doing python setup.py sdist upload. ?There's probably
> a way to use the webui to add the signature after the fact but I haven't
> found a way to sign the existing sdist and upload that signature from the
> command line.

That facility does not exist.

I believe it shouldn't be difficult in the current pypi code to allow
a re-run of the file_upload action (either through the form or through
distutils) where the signature is present and attach the signature to
the existing file, assuming they file accompanying the signature
upload is identical to the existing file.


     Richard

From ubershmekel at gmail.com  Wed Feb  1 08:12:28 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 1 Feb 2012 09:12:28 +0200
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
Message-ID: <CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>

+1 on removing this security loophole in any of the ways suggested here.

Ps I don't think it's "uploaders" vs "downloaders" utility as I'm pretty
sure the uploaders download from pypi as well. And even if it was so,
boosting the trustworthiness of pypi is a win for both sides.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/ebf35efd/attachment.html>

From chris at simplistix.co.uk  Wed Feb  1 09:36:01 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 01 Feb 2012 08:36:01 +0000
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
Message-ID: <4F28F971.60708@simplistix.co.uk>

On 01/02/2012 07:12, Yuval Greenfield wrote:
> +1 on removing this security loophole in any of the ways suggested here.

Good grief, it's not a "security loophole".

If you actually cared about security, you'd already be using, recording 
and checking the MD5 checksums provided with each download and would 
already know that this isn't a security loophole.

If you're not, then quit with the security theater.

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From ubershmekel at gmail.com  Wed Feb  1 10:01:49 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 1 Feb 2012 11:01:49 +0200
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <4F28F971.60708@simplistix.co.uk>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
Message-ID: <CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>

On Wed, Feb 1, 2012 at 10:36 AM, Chris Withers <chris at simplistix.co.uk>wrote:

> On 01/02/2012 07:12, Yuval Greenfield wrote:
>
>> +1 on removing this security loophole in any of the ways suggested here.
>>
>
> Good grief, it's not a "security loophole".
>
> If you actually cared about security, you'd already be using, recording
> and checking the MD5 checksums provided with each download and would
> already know that this isn't a security loophole.
>
> If you're not, then quit with the security theater.
>
> cheers,
>
>
Would you testify that HTTP is secure because I can emulate TLS in
javascript?

PyPI should do what it can within reason to be consistent and safe for all
its users. We're talking about a standard best practice for sites with user
generated content. The original API was aware of this best practice and a
loophole was eventually introduced. Please do read the OP.

"Cheers",

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/edc251b7/attachment.html>

From chris at simplistix.co.uk  Wed Feb  1 10:14:16 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 01 Feb 2012 09:14:16 +0000
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
Message-ID: <4F290268.10204@simplistix.co.uk>

On 01/02/2012 09:01, Yuval Greenfield wrote:
> Would you testify that HTTP is secure because I can emulate TLS in
> javascript?

What's that got to do with the price of eggs?

> PyPI should do what it can within reason to be consistent and safe for
> all its users.

*sigh* that's what the MD5s are for. What threat, exactly are you so 
worried about here? That someone investigates and chooses to use a 
package, and then, having done so, decides to re-download an identical 
version of that package which has been maliciously uploaded, and happens 
to have the same MD5 checksum as the one they've already downloaded?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From richard at python.org  Wed Feb  1 10:15:54 2012
From: richard at python.org (Richard Jones)
Date: Wed, 1 Feb 2012 20:15:54 +1100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <4F28F971.60708@simplistix.co.uk>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
Message-ID: <CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>

On 1 February 2012 19:36, Chris Withers <chris at simplistix.co.uk> wrote:
> If you actually cared about security, you'd already be using, recording and
> checking the MD5 checksums provided with each download and would already
> know that this isn't a security loophole.
>
> If you're not, then quit with the security theater.

I believe the "security theater" of MD5 was proven, and exploits
freely available, back in 2005 :-)


     Richard

From chris at simplistix.co.uk  Wed Feb  1 10:18:41 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 01 Feb 2012 09:18:41 +0000
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
Message-ID: <4F290371.1060303@simplistix.co.uk>

On 01/02/2012 09:15, Richard Jones wrote:
> On 1 February 2012 19:36, Chris Withers<chris at simplistix.co.uk>  wrote:
>> If you actually cared about security, you'd already be using, recording and
>> checking the MD5 checksums provided with each download and would already
>> know that this isn't a security loophole.
>>
>> If you're not, then quit with the security theater.
>
> I believe the "security theater" of MD5 was proven, and exploits
> freely available, back in 2005 :-)

Well now, that's a valid argument, so what hashing technique should we 
be using? ;-)

Chris - https://twitter.com/#!/chrismcdonough/status/159877313771737088

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From mal at egenix.com  Wed Feb  1 10:20:24 2012
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 01 Feb 2012 10:20:24 +0100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
Message-ID: <4F2903D8.3050502@egenix.com>

Richard Jones wrote:
> On 1 February 2012 19:36, Chris Withers <chris at simplistix.co.uk> wrote:
>> If you actually cared about security, you'd already be using, recording and
>> checking the MD5 checksums provided with each download and would already
>> know that this isn't a security loophole.
>>
>> If you're not, then quit with the security theater.
> 
> I believe the "security theater" of MD5 was proven, and exploits
> freely available, back in 2005 :-)

Perhaps we ought to rename the thread to: "Proposal: add SHA hashes to
distribution files", then :-)

I'd be +1 on that since it does actually add security to PyPI.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 01 2012)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From donald.stufft at gmail.com  Wed Feb  1 10:23:11 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 1 Feb 2012 04:23:11 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <4F28F971.60708@simplistix.co.uk>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
Message-ID: <EBBB718DCFFA4A07978338EC282932B5@gmail.com>

It absolutely is.

And I'm already working on a solution that solves the checksum problem for myself. That's all well and good, I won't be affected. But a huge part of the population will still be vulnerable to issues such as previously known code breaking for unknown reasons (which is difficult and infuriating to debug), silent errors that don't actually error things out, but just cause corrupt data (which is worse than it just flat out breaking), or at the far end of the spectrum _can_ be an outright security vulnerability. Pretending like that is outside of the realm of possibilities is irresponsible and wrong.

I prefer to try and protect everyone where we can, especially when the tradeoff is something as relatively minor as needing to create a new version in the rare (or should be rare, if it's not something is very wrong with your release process) that your packaging was bad.(If the problem is with the software itself then it's even more wrong to rerelease it under the same version). So far every solution amounts to either "well then don't use PyPI", or "don't use any of the python packaging tools except for zc.buildout* so that in the rare case that I make a mistake I can be lazy.

* To my knowledge zc.buildout is the only one that supports it. 

On Wednesday, February 1, 2012 at 3:36 AM, Chris Withers wrote:

> On 01/02/2012 07:12, Yuval Greenfield wrote:
> > +1 on removing this security loophole in any of the ways suggested here.
> 
> 
> Good grief, it's not a "security loophole".
> 
> If you actually cared about security, you'd already be using, recording 
> and checking the MD5 checksums provided with each download and would 
> already know that this isn't a security loophole.
> 
> If you're not, then quit with the security theater.
> 
> cheers,
> 
> Chris
> 
> -- 
> Simplistix - Content Management, Batch Processing & Python Consulting
> - http://www.simplistix.co.uk
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/8c07b8d5/attachment.html>

From donald.stufft at gmail.com  Wed Feb  1 10:30:35 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 1 Feb 2012 04:30:35 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <4F2903D8.3050502@egenix.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
	<4F2903D8.3050502@egenix.com>
Message-ID: <1C99F216B4B3466DBF86C7D587EB1179@gmail.com>



On Wednesday, February 1, 2012 at 4:20 AM, M.-A. Lemburg wrote:

> Richard Jones wrote:
> > On 1 February 2012 19:36, Chris Withers <chris at simplistix.co.uk (mailto:chris at simplistix.co.uk)> wrote:
> > > If you actually cared about security, you'd already be using, recording and
> > > checking the MD5 checksums provided with each download and would already
> > > know that this isn't a security loophole.
> > > 
> > > If you're not, then quit with the security theater.
> > 
> > I believe the "security theater" of MD5 was proven, and exploits
> > freely available, back in 2005 :-)
> > 
> 
> 
> Perhaps we ought to rename the thread to: "Proposal: add SHA hashes to
> distribution files", then :-)
> 
> I'd be +1 on that since it does actually add security to PyPI.
This is a similar but doesn't  also good thing to do. IMO it should be sha256, (I would say sha512 but there are slowdown issues on older pythons). 
> 
> -- 
> Marc-Andre Lemburg
> eGenix.com (http://eGenix.com)
> 
> Professional Python Services directly from the Source (#1, Feb 01 2012)
> > > > Python/Zope Consulting and Support ... http://www.egenix.com/
> > > > mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
> > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
> > > > 
> > > 
> > 
> 
> ________________________________________________________________________
> 
> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
> 
> 
> eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> Registered at Amtsgericht Duesseldorf: HRB 46611
> http://www.egenix.com/company/contact/
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/70d52499/attachment-0001.html>

From donald.stufft at gmail.com  Wed Feb  1 10:38:26 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 1 Feb 2012 04:38:26 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <1C99F216B4B3466DBF86C7D587EB1179@gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
	<4F2903D8.3050502@egenix.com>
	<1C99F216B4B3466DBF86C7D587EB1179@gmail.com>
Message-ID: <B7C3DCF4E3814C57B6992CBE86729F49@gmail.com>

ugh it's late, that was meant to say but doesn't cover all situations (and neither does making things write only). But together things would be *a lot* more secure. 


On Wednesday, February 1, 2012 at 4:30 AM, Donald Stufft wrote:

> 
> 
> On Wednesday, February 1, 2012 at 4:20 AM, M.-A. Lemburg wrote:
> 
> > Richard Jones wrote:
> > > On 1 February 2012 19:36, Chris Withers <chris at simplistix.co.uk (mailto:chris at simplistix.co.uk)> wrote:
> > > > If you actually cared about security, you'd already be using, recording and
> > > > checking the MD5 checksums provided with each download and would already
> > > > know that this isn't a security loophole.
> > > > 
> > > > If you're not, then quit with the security theater.
> > > 
> > > I believe the "security theater" of MD5 was proven, and exploits
> > > freely available, back in 2005 :-)
> > > 
> > 
> > 
> > Perhaps we ought to rename the thread to: "Proposal: add SHA hashes to
> > distribution files", then :-)
> > 
> > I'd be +1 on that since it does actually add security to PyPI.
> This is a similar but doesn't  also good thing to do. IMO it should be sha256, (I would say sha512 but there are slowdown issues on older pythons). 
> > 
> > -- 
> > Marc-Andre Lemburg
> > eGenix.com (http://eGenix.com)
> > 
> > Professional Python Services directly from the Source (#1, Feb 01 2012)
> > > > > Python/Zope Consulting and Support ... http://www.egenix.com/
> > > > > mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
> > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
> > > > > 
> > > > 
> > > 
> > 
> > ________________________________________________________________________
> > 
> > ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
> > 
> > 
> > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48
> > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> > Registered at Amtsgericht Duesseldorf: HRB 46611
> > http://www.egenix.com/company/contact/
> > _______________________________________________
> > Catalog-SIG mailing list
> > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> > http://mail.python.org/mailman/listinfo/catalog-sig
> > 
> > 
> > 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/73435ec5/attachment.html>

From richard at python.org  Wed Feb  1 10:42:29 2012
From: richard at python.org (Richard Jones)
Date: Wed, 1 Feb 2012 20:42:29 +1100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <4F290371.1060303@simplistix.co.uk>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
	<4F290371.1060303@simplistix.co.uk>
Message-ID: <CAHrZfZCas9JPgNjcpQAe4-qZTCvWc4bryW=LiUCrSuNmJ7zHjQ@mail.gmail.com>

On 1 February 2012 20:18, Chris Withers <chris at simplistix.co.uk> wrote:
> Chris - https://twitter.com/#!/chrismcdonough/status/159877313771737088

Oh FFS

From richard at python.org  Wed Feb  1 10:43:27 2012
From: richard at python.org (Richard Jones)
Date: Wed, 1 Feb 2012 20:43:27 +1100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZCas9JPgNjcpQAe4-qZTCvWc4bryW=LiUCrSuNmJ7zHjQ@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
	<4F290371.1060303@simplistix.co.uk>
	<CAHrZfZCas9JPgNjcpQAe4-qZTCvWc4bryW=LiUCrSuNmJ7zHjQ@mail.gmail.com>
Message-ID: <CAHrZfZBEm-q4tR=_=3m3V=EexDgBAUh+_-KgCJtHg3MBFJ9U+g@mail.gmail.com>

I'll just point out that my initial email said nothing about security.

From donald.stufft at gmail.com  Wed Feb  1 10:44:52 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 1 Feb 2012 04:44:52 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZBEm-q4tR=_=3m3V=EexDgBAUh+_-KgCJtHg3MBFJ9U+g@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CAHrZfZCmJLrCvza6MiQVViuG_P1e_xg9m037oDDJD-swLnaxxQ@mail.gmail.com>
	<4F290371.1060303@simplistix.co.uk>
	<CAHrZfZCas9JPgNjcpQAe4-qZTCvWc4bryW=LiUCrSuNmJ7zHjQ@mail.gmail.com>
	<CAHrZfZBEm-q4tR=_=3m3V=EexDgBAUh+_-KgCJtHg3MBFJ9U+g@mail.gmail.com>
Message-ID: <59D529A3DF3249828848ADD26591DA6F@gmail.com>

No I was the one who said that ;) But it's still a valid complaint imo.

On Wednesday, February 1, 2012 at 4:43 AM, Richard Jones wrote:

> I'll just point out that my initial email said nothing about security.
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/cef3b917/attachment.html>

From ubershmekel at gmail.com  Wed Feb  1 12:06:18 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 1 Feb 2012 13:06:18 +0200
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <4F290268.10204@simplistix.co.uk>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
Message-ID: <CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>

On Wed, Feb 1, 2012 at 11:14 AM, Chris Withers <chris at simplistix.co.uk>wrote:

> On 01/02/2012 09:01, Yuval Greenfield wrote:
>
>> Would you testify that HTTP is secure because I can emulate TLS in
>> javascript?
>>
>
> What's that got to do with the price of eggs?
>
>
>
I can maintain my own personal log of package SHA-512's and thus locally
avoid this security hole in PyPI. The system as a whole is still vulnerable
by default. That's why HTTP is considered insecure even though you can
build secure solutions on top of it. PyPI would be the insecure
infrastructure upon which secure frameworks can be built. Immutability will
make the default behavior of pypi more secure.



>  PyPI should do what it can within reason to be consistent and safe for
>> all its users.
>>
>
> *sigh* that's what the MD5s are for. What threat, exactly are you so
> worried about here? That someone investigates and chooses to use a package,
> and then, having done so, decides to re-download an identical version of
> that package which has been maliciously uploaded, and happens to have the
> same MD5 checksum as the one they've already downloaded?
>
>
Let's assume I made a package called pybanker that requires a specific
version of SQLAlchemy (eg 0.6.8). When I tell people to
download SQLAlchemy 0.6.8 do I have to tell them the exact hash? Does the
setup.py/cfg allow me to require a specific hash on SQLAlchemy when
automatically resolving dependencies in pip/easy_install? So now when banks
around the world are going to use pybanker and thus SQLAlchemy 0.6.8 - they
don't know what was the original hash. In the meantime a security threat
has manifested in sqlalchemy (eg pythonpackages was hacked, an sqlalchemy
maintainer password/computer/network was compromised, etc). The hacker
modifies SQLAlchemy 0.6.8 to work perfectly while adding a backdoor to the
system or relaying all transactions to a remote server.

I hope we don't have to wait for this attack vector to be used (and
detected and publicized) before this loophole is patched.

Obviously this isn't the only problem if the account of an SQLAlchemy
maintainer is compromised - other threats can manifest as well. That
doesn't mean this specific threat should be ignored, especially considering
that it's a stealthy vector.

tl;dr - the classic bait and switch is why user generated content is
immutable on most web services. If edits are allowed they are usually only
so for a limited amount of time or require an administrator in the loop.

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/fd5ddc26/attachment.html>

From donald.stufft at gmail.com  Wed Feb  1 12:10:49 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 1 Feb 2012 06:10:49 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
Message-ID: <7347521339674C329D59F2725A901C59@gmail.com>

I should mention that this scenario is the worst case scenario, and isn't the likely one. However I think the possible damages from it well outweigh the small amount of benefit from mutable packages. 

The more likely scenarios (on the failure side) are either applications breaking upon install/deploy or silently corrupting data. Both of which, but especially the silently corrupting data case I think the possible damages again outweigh the small benefit from mutable packages. 


On Wednesday, February 1, 2012 at 6:06 AM, Yuval Greenfield wrote:

> On Wed, Feb 1, 2012 at 11:14 AM, Chris Withers <chris at simplistix.co.uk (mailto:chris at simplistix.co.uk)> wrote:
> > On 01/02/2012 09:01, Yuval Greenfield wrote:
> > > Would you testify that HTTP is secure because I can emulate TLS in
> > > javascript?
> > 
> > What's that got to do with the price of eggs?
> > 
> > 
> 
> I can maintain my own personal log of package SHA-512's and thus locally avoid this security hole in PyPI. The system as a whole is still vulnerable by default. That's why HTTP is considered insecure even though you can build secure solutions on top of it. PyPI would be the insecure infrastructure upon which secure frameworks can be built. Immutability will make the default behavior of pypi more secure. 
> 
>  
> > > PyPI should do what it can within reason to be consistent and safe for
> > > all its users.
> > 
> > *sigh* that's what the MD5s are for. What threat, exactly are you so worried about here? That someone investigates and chooses to use a package, and then, having done so, decides to re-download an identical version of that package which has been maliciously uploaded, and happens to have the same MD5 checksum as the one they've already downloaded?
> > 
> 
> Let's assume I made a package called pybanker that requires a specific version of SQLAlchemy (eg 0.6.8). When I tell people to download SQLAlchemy 0.6.8 do I have to tell them the exact hash? Does the setup.py/cfg (http://setup.py/cfg) allow me to require a specific hash on SQLAlchemy when automatically resolving dependencies in pip/easy_install? So now when banks around the world are going to use pybanker and thus SQLAlchemy 0.6.8 - they don't know what was the original hash. In the meantime a security threat has manifested in sqlalchemy (eg pythonpackages was hacked, an sqlalchemy maintainer password/computer/network was compromised, etc). The hacker modifies SQLAlchemy 0.6.8 to work perfectly while adding a backdoor to the system or relaying all transactions to a remote server. 
> 
> I hope we don't have to wait for this attack vector to be used (and detected and publicized) before this loophole is patched.
> 
> Obviously this isn't the only problem if the account of an SQLAlchemy maintainer is compromised - other threats can manifest as well. That doesn't mean this specific threat should be ignored, especially considering that it's a stealthy vector. 
> 
> tl;dr - the classic bait and switch is why user generated content is immutable on most web services. If edits are allowed they are usually only so for a limited amount of time or require an administrator in the loop. 
> 
> Yuval 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/8db024f0/attachment.html>

From solipsis at pitrou.net  Wed Feb  1 15:29:11 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 1 Feb 2012 14:29:11 +0000 (UTC)
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
Message-ID: <loom.20120201T152346-808@post.gmane.org>

Yuval Greenfield <ubershmekel <at> gmail.com> writes:
> 
> Obviously this isn't the only problem if the account of an SQLAlchemy
> maintainer is compromised - other threats can manifest as well.

So, why you think PyPI has to have protections against the hacking of
maintainers' accounts is beyond me. That's a completely unreasonable
expectation.

Besides, being able to delete a release is mandatory (imagine you have uploaded
confidential files by mistake).

I don't even understand why people are having this discussion. PyPI is not a
packaging *authority*. It's not Debian or Fedora or anything like that. It's
just a place for people to publish files and metadata. You can't trust it any
more than you can trust the uploaders themselves.

Regards

Antoine.



From ubershmekel at gmail.com  Wed Feb  1 15:52:43 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 1 Feb 2012 16:52:43 +0200
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <loom.20120201T152346-808@post.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<loom.20120201T152346-808@post.gmane.org>
Message-ID: <CANSw7KzFd6Jnpe_2zmGLO=6J6NfhZkUfQ4crgRLGQ3fswQA3GA@mail.gmail.com>

On Wed, Feb 1, 2012 at 4:29 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Yuval Greenfield <ubershmekel <at> gmail.com> writes:
> >
> > Obviously this isn't the only problem if the account of an SQLAlchemy
> > maintainer is compromised - other threats can manifest as well.
>
> So, why you think PyPI has to have protections against the hacking of
> maintainers' accounts is beyond me. That's a completely unreasonable
> expectation.
>
> Besides, being able to delete a release is mandatory (imagine you have
> uploaded
> confidential files by mistake).
>
>

The original proposal was "retaining a record of the uploaded file (though
not the contents) so that future uploads with the same name wouldn't be
allowed."

It sounds like you would be happy with that proposal.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/06f70efc/attachment-0001.html>

From donald.stufft at gmail.com  Wed Feb  1 15:54:29 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 1 Feb 2012 09:54:29 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <loom.20120201T152346-808@post.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<loom.20120201T152346-808@post.gmane.org>
Message-ID: <F7FDFB48E7B94F9BB2887F2733CB5E15@gmail.com>



On Wednesday, February 1, 2012 at 9:29 AM, Antoine Pitrou wrote:

> Yuval Greenfield <ubershmekel <at> gmail.com (http://gmail.com)> writes:
> > 
> > Obviously this isn't the only problem if the account of an SQLAlchemy
> > maintainer is compromised - other threats can manifest as well.
> > 
> 
> 
> So, why you think PyPI has to have protections against the hacking of
> maintainers' accounts is beyond me. That's a completely unreasonable
> expectation.
> 
> 

That's only one relatively unlikely scenario where this would be useful and a good change, there are other more likely scenarios where this would also be a good change.
> 
> Besides, being able to delete a release is mandatory (imagine you have uploaded
> confidential files by mistake).
> 
> 

Nothing in this proposal removes the ability to delete files. You just won't be able to re upload a file of the same name (basically version). So if you accidentally include confidential files in version 2.3, you can delete version 2.3, but you'll have to release the fixed version as 2.3.1. 
> 
> I don't even understand why people are having this discussion. PyPI is not a
> packaging *authority*. It's not Debian or Fedora or anything like that. It's
> just a place for people to publish files and metadata. You can't trust it any
> more than you can trust the uploaders themselves.
> 
> 

Semantics arguments are boring and tired. People depend on PyPI and the packages installed there. They depend on the ability to pin to a specific tested release of libraries and they should be able to depend on the fact that if they ask for version 1.1 of library XYZ they will always get the exact same package.

What if python.org decided to replace the download links for Python 2.7.2 with a new version of Python 2.7.2 with new bugs fixed, or maybe a typo? What if those "harmless" fixes broke my software because I was depending on that behavior and now my software just stops working. For no reason what so ever. What's worse is it still works on some computers (where I have the _original_ version installed) but on other computers it just doesn't work.
> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/23f685c1/attachment.html>

From fuzzyman at gmail.com  Wed Feb  1 16:03:44 2012
From: fuzzyman at gmail.com (Michael Foord)
Date: Wed, 1 Feb 2012 15:03:44 +0000
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <F7FDFB48E7B94F9BB2887F2733CB5E15@gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<loom.20120201T152346-808@post.gmane.org>
	<F7FDFB48E7B94F9BB2887F2733CB5E15@gmail.com>
Message-ID: <CAKCKLWzAdOq0xO6SNdB7qkyzVAJvzjKYxeo+=Cd7U0uLxHMW8A@mail.gmail.com>

On 1 February 2012 14:54, Donald Stufft <donald.stufft at gmail.com> wrote:

>
>  On Wednesday, February 1, 2012 at 9:29 AM, Antoine Pitrou wrote:
>
> Yuval Greenfield <ubershmekel <at> gmail.com> writes:
>
>
> Obviously this isn't the only problem if the account of an SQLAlchemy
> maintainer is compromised - other threats can manifest as well.
>
>
> So, why you think PyPI has to have protections against the hacking of
> maintainers' accounts is beyond me. That's a completely unreasonable
> expectation.
>
> That's only one relatively unlikely scenario where this would be useful
> and a good change, there are other more likely scenarios where this would
> also be a good change.
>
>
> Besides, being able to delete a release is mandatory (imagine you have
> uploaded
> confidential files by mistake).
>
> Nothing in this proposal removes the ability to delete files. You just
> won't be able to re upload a file of the same name (basically version). So
> if you accidentally include confidential files in version 2.3, you can
> delete version 2.3, but you'll have to release the fixed version as 2.3.1.
>

Which can be a major hassle, even with automated release procedures
(announcements, links, etc).


>
> I don't even understand why people are having this discussion. PyPI is not
> a
> packaging *authority*. It's not Debian or Fedora or anything like that.
> It's
> just a place for people to publish files and metadata. You can't trust it
> any
> more than you can trust the uploaders themselves.
>
> Semantics arguments are boring and tired. People depend on PyPI and the
> packages installed there. They depend on the ability to pin to a specific
> tested release of libraries and they should be able to depend on the fact
> that if they ask for version 1.1 of library XYZ they will always get the
> exact same package.
>
> What if python.org decided to replace the download links for Python 2.7.2
> with a new version of Python 2.7.2 with new bugs fixed, or maybe a typo?
> What if those "harmless" fixes broke my software because I was depending on
> that behavior and now my software just stops working. For no reason what so
> ever. What's worse is it still works on some computers (where I have the
> _original_ version installed) but on other computers it just doesn't work.
>

There is a big difference however. Python.org is a distributor of software
and we promise not to do that. PyPI is not a single distributor with
distribution policies - it is a platform for individual package authors to
provide downloads, with their own distribution practises and policies.

The security features you want from pypi can already be had by pinning
secure hashes to distribution versions.

Feel free to carry on arguing. Fortunately when there is a controversial
issue at stake with no consensus (several strong +1 and several strong -1
in this case), the status quo wins. Absent a BDFL decision of course. ;-)

All the best,

Michael Foord




>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/efe68733/attachment.html>

From solipsis at pitrou.net  Wed Feb  1 16:18:40 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 1 Feb 2012 15:18:40 +0000 (UTC)
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<loom.20120201T152346-808@post.gmane.org>
	<F7FDFB48E7B94F9BB2887F2733CB5E15@gmail.com>
Message-ID: <loom.20120201T155824-230@post.gmane.org>

Donald Stufft <donald.stufft <at> gmail.com> writes:
> I don't even understand why people are having this discussion. PyPI is not a
> packaging *authority*. It's not Debian or Fedora or anything like that. It's
> just a place for people to publish files and metadata. You can't trust it any
> more than you can trust the uploaders themselves.
> 
> Semantics arguments are boring and tired.

Just because you don't understand them doesn't make them irrelevant.
PyPI is *not* secure. Any maintainer can upload whatever (s)he wants. You are
asking for a fix that won't do any good for the general problem.

> People depend on PyPI and the packages installed there. They depend on the
> ability to pin to a specific tested release of libraries and they should be
> able to depend on the fact that if they ask for version 1.1 of library XYZ
> they will always get the exact same package.

Are you sure you will get the "exact same package"? What if the Linux version
has different contents from the Windows version? Or the py-2.6 version was not
built properly (while the py-2.7 version was)?
Perhaps there was originally only a source release, and the attacker added some
download links for malicious binary builds?

> What if python.org decided to replace the download links for Python 2.7.2
> with a new version of Python 2.7.2 with new bugs fixed, or maybe a typo?

What if? That may be a good reason to stop trusting python.org.
Similarly, if a maintainer of a 3rd-party package uploads a significantly
different file for a given release, you should perhaps stop trusting them too.
But *you* must make that decision. You can't ask an automated software system to
solve trust issues for you.

> What if those "harmless" fixes broke my software because I was depending on
> that behavior and now my software just stops working.

What if? The right attitude is certainly not to complain to PyPI. Instead,
complain to the maintainer.

Regards

Antoine.



From donald.stufft at gmail.com  Wed Feb  1 16:31:03 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 1 Feb 2012 10:31:03 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <loom.20120201T155824-230@post.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<loom.20120201T152346-808@post.gmane.org>
	<F7FDFB48E7B94F9BB2887F2733CB5E15@gmail.com>
	<loom.20120201T155824-230@post.gmane.org>
Message-ID: <E686B0F21D2B4B0E9D23F6F4F74AE582@gmail.com>



On Wednesday, February 1, 2012 at 10:18 AM, Antoine Pitrou wrote:

> Donald Stufft <donald.stufft <at> gmail.com (http://gmail.com)> writes:
> > I don't even understand why people are having this discussion. PyPI is not a
> > packaging *authority*. It's not Debian or Fedora or anything like that. It's
> > just a place for people to publish files and metadata. You can't trust it any
> > more than you can trust the uploaders themselves.
> > 
> > Semantics arguments are boring and tired.
> 
> Just because you don't understand them doesn't make them irrelevant.
> PyPI is *not* secure. Any maintainer can upload whatever (s)he wants. You are
> asking for a fix that won't do any good for the general problem.
> 
> 

No, them being irrelevant makes them irrelevant. 
> 
> > People depend on PyPI and the packages installed there. They depend on the
> > ability to pin to a specific tested release of libraries and they should be
> > able to depend on the fact that if they ask for version 1.1 of library XYZ
> > they will always get the exact same package.
> > 
> 
> 
> Are you sure you will get the "exact same package"? What if the Linux version
> has different contents from the Windows version? Or the py-2.6 version was not
> built properly (while the py-2.7 version was)?
> Perhaps there was originally only a source release, and the attacker added some
> download links for malicious binary builds?
> 
> > What if python.org (http://python.org) decided to replace the download links for Python 2.7.2
> > with a new version of Python 2.7.2 with new bugs fixed, or maybe a typo?
> > 
> 
> 
> What if? That may be a good reason to stop trusting python.org (http://python.org).
> Similarly, if a maintainer of a 3rd-party package uploads a significantly
> different file for a given release, you should perhaps stop trusting them too.
> But *you* must make that decision. You can't ask an automated software system to
> solve trust issues for you.
> 
> > What if those "harmless" fixes broke my software because I was depending on
> > that behavior and now my software just stops working.
> > 
> 
> 
> What if? The right attitude is certainly not to complain to PyPI. Instead,
> complain to the maintainer.
> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/51d4051b/attachment.html>

From ubershmekel at gmail.com  Wed Feb  1 16:34:20 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 1 Feb 2012 17:34:20 +0200
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <loom.20120201T155824-230@post.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<loom.20120201T152346-808@post.gmane.org>
	<F7FDFB48E7B94F9BB2887F2733CB5E15@gmail.com>
	<loom.20120201T155824-230@post.gmane.org>
Message-ID: <CANSw7KxV2pKX0S5cLQf1-=L8VNPyuGAtmON6wiHjFa7BWVPsBw@mail.gmail.com>

On Wed, Feb 1, 2012 at 5:18 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Donald Stufft <donald.stufft <at> gmail.com> writes:
> > I don't even understand why people are having this discussion. PyPI is
> not a
> > packaging *authority*. It's not Debian or Fedora or anything like that.
> It's
> > just a place for people to publish files and metadata. You can't trust
> it any
> > more than you can trust the uploaders themselves.
> >
> > Semantics arguments are boring and tired.
>
> Just because you don't understand them doesn't make them irrelevant.
> PyPI is *not* secure. Any maintainer can upload whatever (s)he wants. You
> are
> asking for a fix that won't do any good for the general problem.
>
[...]
> Regards
>
> Antoine.
>
>
>
With that attitude you must really hate bumping release versions.

Anyhow, it's a simple best practice that was the original design of the
system. As was mentioned, of course there are more vulnerabilities.
Improving the system one part at a time would still be a good idea.

This feature would be a big win in security and sanity for a very small
cost of convenience for very rare occasions and needs.

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/18f31a15/attachment.html>

From pje at telecommunity.com  Wed Feb  1 23:57:08 2012
From: pje at telecommunity.com (PJ Eby)
Date: Wed, 1 Feb 2012 17:57:08 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
Message-ID: <CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>

On Wed, Feb 1, 2012 at 6:06 AM, Yuval Greenfield <ubershmekel at gmail.com>wrote:

> Does the setup.py/cfg allow me to require a specific hash on SQLAlchemy
> when automatically resolving dependencies in pip/easy_install?
>

Yes, at least for easy_install.  You tack on  #md5=.... to your find_links
URLs, and specify an exact version.  easy_install will refuse to install
them if the MD5 doesn't match.  (This will work better for source packages
than binaries, of course, since you'd only need to include one link and MD5
signature in that case.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120201/9816092e/attachment.html>

From richard at python.org  Thu Feb  2 02:36:29 2012
From: richard at python.org (Richard Jones)
Date: Thu, 2 Feb 2012 12:36:29 +1100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
Message-ID: <CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>

Summarising the responses:

8 at +1
3 at -1

Several posts with no stated positions.

Given it appears to be controversial, I'm just going to drop it. I
just don't need the aggravation. PyPI can retain its ability to serve
up potentially confusing file content.


    Richard

From djay at pretaweb.com  Thu Feb  2 03:59:29 2012
From: djay at pretaweb.com (Dylan Jay)
Date: Thu, 2 Feb 2012 13:59:29 +1100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
	<CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>
Message-ID: <2D054A29-A6BD-49A9-96DF-498AEE4EB618@pretaweb.com>

It it counts I'm +1.
The problem with counting +1 or -1 is it assumes the silent majority  
would be similarly split in opinion.

---
Dylan Jay
Technical Solutions Manager
PretaWeb: Multisite Performance Support
P: +612 80819071 | M: +61421477460 | twitter.com/djay75 | linkedin.com/ 
in/djay75

On 02/02/2012, at 12:36 PM, Richard Jones wrote:

> Summarising the responses:
>
> 8 at +1
> 3 at -1
>
> Several posts with no stated positions.
>
> Given it appears to be controversial, I'm just going to drop it. I
> just don't need the aggravation. PyPI can retain its ability to serve
> up potentially confusing file content.
>
>
>    Richard
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig


From ben+python at benfinney.id.au  Thu Feb  2 07:00:50 2012
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 02 Feb 2012 17:00:50 +1100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
	<CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>
Message-ID: <87obthwz8t.fsf@benfinney.id.au>

Richard Jones <richard at python.org> writes:

> Given it appears to be controversial, I'm just going to drop it. I
> just don't need the aggravation. PyPI can retain its ability to serve
> up potentially confusing file content.

+1 for refusing new uploads of different files served with old names.

Like it or not, PyPI is now a dependency system for applications, and
its API relies on filenames. I don't want the aggravation of a
dependency API that relies on filenames, but allows the same name to
serve a different file.

-- 
 \       ?Science and religion are incompatible in the same sense that |
  `\       the serious pursuit of knowledge of reality is incompatible |
_o__)                       with bullshit.? ?Paul Z. Myers, 2010-03-14 |
Ben Finney


From fuzzyman at gmail.com  Thu Feb  2 14:01:01 2012
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 2 Feb 2012 13:01:01 +0000
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
	<CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>
Message-ID: <CAKCKLWwuK9Ac+rjWzWB9pL_NbJufEHe9TreTXkMn0thX16E+Gw@mail.gmail.com>

On 2 February 2012 01:36, Richard Jones <richard at python.org> wrote:

> Summarising the responses:
>
> 8 at +1
> 3 at -1
>
> Several posts with no stated positions.
>


Several posts with no explicit -1, but I see objections/misgivings from the
following:

Me
Martin Loewis
Phillip Eby
Antoine Pitrou
Robert Collins
MA Lemburg

Plus Chris Withers sceptical of the "security" advantages, although not
explicitly objecting.

Note that even if this hole is plugged it still offers no security
advantage to users of tools like pip/easy_install - all a package
maintainer has to do is switch to hosting the download themselves and the
tools will still merrily install the specified version from wherever it is
hosted (using the download link from pypi). So the *only* security fix is
to specify a secure hash to the install tool, not screw over package
maintainers with more restrictions on pypi.

Given the issues with md5, adding SHA (or similar) hashes to pypi would be
a much better use of time (IMO).

All the best,

Michael Foord



>
> Given it appears to be controversial, I'm just going to drop it. I
> just don't need the aggravation. PyPI can retain its ability to serve
> up potentially confusing file content.
>
>
>    Richard
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120202/0d4d9744/attachment.html>

From chris at simplistix.co.uk  Thu Feb  2 14:40:23 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 02 Feb 2012 13:40:23 +0000
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAKCKLWwuK9Ac+rjWzWB9pL_NbJufEHe9TreTXkMn0thX16E+Gw@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
	<CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>
	<CAKCKLWwuK9Ac+rjWzWB9pL_NbJufEHe9TreTXkMn0thX16E+Gw@mail.gmail.com>
Message-ID: <4F2A9247.7070703@simplistix.co.uk>

On 02/02/2012 13:01, Michael Foord wrote:
>
> Plus Chris Withers sceptical of the "security" advantages, although not
> explicitly objecting.

I'm -0.

I don't see the point of removing flexibility, but I can see the 
argument of helping package authors being less considerate.

I just get grumpy when people pretend things are about security that 
really aren't  ;-)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From carl at oddbird.net  Thu Feb  2 16:55:31 2012
From: carl at oddbird.net (Carl Meyer)
Date: Thu, 02 Feb 2012 08:55:31 -0700
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>	<jga5ff$so0$1@dough.gmane.org>
	<20120201020841.GF23782@unaka.lan>	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>	<4F28F971.60708@simplistix.co.uk>	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>	<4F290268.10204@simplistix.co.uk>	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
Message-ID: <4F2AB1F3.8010500@oddbird.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/01/2012 03:57 PM, PJ Eby wrote:
> On Wed, Feb 1, 2012 at 6:06 AM, Yuval Greenfield <ubershmekel at gmail.com
> <mailto:ubershmekel at gmail.com>> wrote:
> 
>     Does the setup.py/cfg <http://setup.py/cfg> allow me to require a
>     specific hash on SQLAlchemy when automatically resolving
>     dependencies in pip/easy_install?
> 
> 
> Yes, at least for easy_install.  You tack on  #md5=.... to your
> find_links URLs, and specify an exact version.  easy_install will refuse
> to install them if the MD5 doesn't match.  (This will work better for
> source packages than binaries, of course, since you'd only need to
> include one link and MD5 signature in that case.)

FWIW, the exact same technique works if you install with pip.

I haven't been following the conversation closely, but I thought I saw
an assertion or two go by that pip doesn't check MD5 hashes of PyPI
downloads. This is not true; it always checks them by default, assuming
the download link includes the md5 hash fragment (which PyPI-hosted
downloads always do).

If you want to assert a specific md5 hash in your requirements file,
you'd need to link to the sdist directly (rather than using
packagename==version) and include the #md5= hash fragment in the link.

Carl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8qsfMACgkQ8W4rlRKtE2f+NACeM3KxyXNZ3DrHclawtckxc5iT
5d0AnR6ClIyCTz9eJGQtio69mSAOuHtB
=3szU
-----END PGP SIGNATURE-----

From martin at v.loewis.de  Thu Feb  2 19:55:40 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 02 Feb 2012 19:55:40 +0100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <2D054A29-A6BD-49A9-96DF-498AEE4EB618@pretaweb.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>	<jga5ff$so0$1@dough.gmane.org>
	<20120201020841.GF23782@unaka.lan>	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>	<4F28F971.60708@simplistix.co.uk>	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>	<4F290268.10204@simplistix.co.uk>	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>	<CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>	<CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>
	<2D054A29-A6BD-49A9-96DF-498AEE4EB618@pretaweb.com>
Message-ID: <4F2ADC2C.9010908@v.loewis.de>

Am 02.02.2012 03:59, schrieb Dylan Jay:
> It it counts I'm +1.
> The problem with counting +1 or -1 is it assumes the silent majority
> would be similarly split in opinion.

I think Richard is following a "in case of significant opposition to
change, preserve the status quo" here. There have been cases of "but
I'm sure I know better than the opposition" in PyPI's history, and
some of these ended up with a debacle, hateful email, and so on
(I know, because I *did* know better than the opposition :-)

If you follow the rule Richard has exercised, the number of supporters
must be *much* larger than the opposition, so that the opposition
can be considered insignificant.

Regards,
Martin

From solipsis at pitrou.net  Fri Feb  3 01:24:36 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 3 Feb 2012 00:24:36 +0000 (UTC)
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<jga5ff$so0$1@dough.gmane.org> <20120201020841.GF23782@unaka.lan>
	<CAHrZfZBhS6TBnrRtbh4AwQnpBMaDHDjardCs60NC5VnO4ag0dw@mail.gmail.com>
	<CANSw7KyzgqXDx_D6wuv89NLx_3+5iD2-Xu-CDZEy_wi2xvNVUA@mail.gmail.com>
	<4F28F971.60708@simplistix.co.uk>
	<CANSw7KwSn0iZ+dkVCB7DF1hAHWY2ceY4CTz2qQKMx8QUFVY4Zw@mail.gmail.com>
	<4F290268.10204@simplistix.co.uk>
	<CANSw7KyfY_C2F7S2KenZFZ-StuCxajm1H5qVEScnZjB8Hs0yVA@mail.gmail.com>
	<CALeMXf4-KcOJGSCoVEqCm=Lb_dk5ihabqoF6n-CrUXpqSJ6Trw@mail.gmail.com>
	<CAHrZfZAL588gzaxVtOZFBAdUeTrscMNL6RDWZJN_n0UbrggAOA@mail.gmail.com>
	<CAKCKLWwuK9Ac+rjWzWB9pL_NbJufEHe9TreTXkMn0thX16E+Gw@mail.gmail.com>
Message-ID: <loom.20120203T011902-233@post.gmane.org>

Michael Foord <fuzzyman <at> gmail.com> writes:
> 
> Given the issues with md5, adding SHA (or similar) hashes to pypi would
> be a much better use of time (IMO).

Also, implement http://www.python.org/dev/peps/pep-0381/#mirror-authenticity

Regards

Antoine.



From donald.stufft at gmail.com  Fri Feb  3 06:39:48 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Fri, 3 Feb 2012 00:39:48 -0500
Subject: [Catalog-sig] Proposal: Additional PyPI Mirror DNS aliases
Message-ID: <FB1F7BD1809A43569A10FF0FECBCCC8A@gmail.com>

Currently there is one DNS name called last.pypi.python.org where the last mirror in alphabetical order is returned. 

I think that it would be very useful to provide an additional 2 DNS names.

random.pypi.python.org - This would just be a round robin to all of the mirrors, the goal being to (theoretically) evenly distributing the incoming requests among them.

fresh.pypi.python.org - This would point towards the mirror that last synchronized with PyPI. The goal being that people can use this to have a higher chance of having all of the packages synced with PyPI (since the other mirrors have had a longer time since sync, they will have a larger window for having missed packages/versions). 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120203/50cccb91/attachment-0001.html>

From martin at v.loewis.de  Fri Feb  3 22:56:51 2012
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 03 Feb 2012 22:56:51 +0100
Subject: [Catalog-sig] Proposal: Additional PyPI Mirror DNS aliases
In-Reply-To: <FB1F7BD1809A43569A10FF0FECBCCC8A@gmail.com>
References: <FB1F7BD1809A43569A10FF0FECBCCC8A@gmail.com>
Message-ID: <4F2C5823.2060108@v.loewis.de>

> random.pypi.python.org - This would just be a round robin to all of the
> mirrors, the goal being to (theoretically) evenly distributing the
> incoming requests among them.

It's less useful as it sounds. Clients are supposed to check whether the
mirror they use is out of date. If they get a random mirror, they may
end up with an unusable one.

> fresh.pypi.python.org - This would point towards the mirror that last
> synchronized with PyPI. The goal being that people can use this to have
> a higher chance of having all of the packages synced with PyPI (since
> the other mirrors have had a longer time since sync, they will have a
> larger window for having missed packages/versions).

That's not implementable using our current DNS hosting service (I'm
not sure that round-robin is implementable even).

Regards,
Martin

From stefan-usenet at bytereef.org  Sat Feb  4 20:40:54 2012
From: stefan-usenet at bytereef.org (Stefan Krah)
Date: Sat, 4 Feb 2012 20:40:54 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
Message-ID: <20120204194054.GA29629@sleipnir.bytereef.org>

Hello,

I wonder what the point of a site like pythonpackages.com is. For my
package (cdecimal) it spreads misinformation on two counts:

  1) The number of downloads cannot be established since cdecimal
     is not hosted on PyPI.

  2) "Now supports Python 3!" is false since cdecimal has supported
     Python3 from the very beginning.


Otherwise it displays Google ads and the same information as PyPI,
only in a less readable manner.


Stefan Krah




From faassen at startifact.com  Sat Feb  4 21:09:48 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Sat, 04 Feb 2012 21:09:48 +0100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
Message-ID: <jgk3ac$1e5$1@dough.gmane.org>

On 01/30/2012 12:47 AM, Richard Jones wrote:

> Your thoughts?

+1, though I'd advocate a "grace period" of a few hours after initial 
placement.

Regards,

Martijn




From faassen at startifact.com  Sat Feb  4 21:12:29 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Sat, 04 Feb 2012 21:12:29 +0100
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <jga1lb$66k$1@dough.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
	<FA61D66685314D12A150635916BB50FD@gmail.com>
	<jga1lb$66k$1@dough.gmane.org>
Message-ID: <jgk3fd$1e5$2@dough.gmane.org>

On 02/01/2012 01:40 AM, Terry Reedy wrote:
> On 1/31/2012 6:43 PM, Donald Stufft wrote:
>> I don't think anyone is arguing that it's not occasionally useful. The
>> question to answer is the occasional usefulness worth the risks that
>> come with it. In my opinion the small utility (being able to correct a
>> borked packaging job) is not worth the risks to both my applications
>> stability, and the security of my entire system.
>
> The question is whether, on each issue, PyPI should be optimized for
> authors (who provide their modules for free) or for users. Both choices
> are defensible. However, if all choices are made in favor of users,
> there will very likely be fewer things uploaded or even listed, which is
> not favorable for users.

I don't think it's a simple dichotomy. If the authors follow certain 
best practices they might retain more users, say. And if system is great 
for users and has lots of them, that motivates authors to work with it.

Regards,

Martijn



From jim at zope.com  Sat Feb  4 22:01:04 2012
From: jim at zope.com (Jim Fulton)
Date: Sat, 4 Feb 2012 16:01:04 -0500
Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole
In-Reply-To: <jgk3fd$1e5$2@dough.gmane.org>
References: <CAHrZfZAd_mrfQ6ScV+rq52q9pmeEhMVRiP4_GSgyXhrQuY2ymg@mail.gmail.com>
	<CAKCKLWybc-v8xpzm+fdBmH-HaeSAOFAPWLiassG-hSSU5+KKHA@mail.gmail.com>
	<FA61D66685314D12A150635916BB50FD@gmail.com>
	<jga1lb$66k$1@dough.gmane.org> <jgk3fd$1e5$2@dough.gmane.org>
Message-ID: <CAPDm-FisivkbuwFopfo-7G9XS+ZPGmtYzE4gYRAVVAoNWQ5CgA@mail.gmail.com>

On Sat, Feb 4, 2012 at 3:12 PM, Martijn Faassen <faassen at startifact.com> wrote:
> On 02/01/2012 01:40 AM, Terry Reedy wrote:
>>
>> On 1/31/2012 6:43 PM, Donald Stufft wrote:
>>>
>>> I don't think anyone is arguing that it's not occasionally useful. The
>>> question to answer is the occasional usefulness worth the risks that
>>> come with it. In my opinion the small utility (being able to correct a
>>> borked packaging job) is not worth the risks to both my applications
>>> stability, and the security of my entire system.
>>
>>
>> The question is whether, on each issue, PyPI should be optimized for
>> authors (who provide their modules for free) or for users. Both choices
>> are defensible. However, if all choices are made in favor of users,
>> there will very likely be fewer things uploaded or even listed, which is
>> not favorable for users.
>
>
> I don't think it's a simple dichotomy. If the authors follow certain best
> practices they might retain more users, say. And if system is great for
> users and has lots of them, that motivates authors to work with it.

Well said. Thanks.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From kai.diefenbach at iqpp.de  Sun Feb  5 14:37:01 2012
From: kai.diefenbach at iqpp.de (Kai Diefenbach)
Date: Sun, 5 Feb 2012 14:37:01 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
References: <20120204194054.GA29629@sleipnir.bytereef.org>
Message-ID: <jgm0lt$8bl$1@dough.gmane.org>

Hi,

On 2012-02-04 19:40:54 +0000, Stefan Krah said:

>   1) The number of downloads cannot be established since cdecimal
>      is not hosted on PyPI.

Why not? Packages, which are not hosted on PyPi suck.

Kai



From tjreedy at udel.edu  Sun Feb  5 19:12:53 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 05 Feb 2012 13:12:53 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgm0lt$8bl$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org>
Message-ID: <jgmgrf$ndi$1@dough.gmane.org>

On 2/5/2012 8:37 AM, Kai Diefenbach wrote:
> Why not? Packages, which are not hosted on PyPi suck.

This is a technical discussion list, not a flame list.
That comment is both wrong and unhelpful.

-- 
Terry Jan Reedy


From aclark at aclark.net  Sun Feb  5 19:48:17 2012
From: aclark at aclark.net (Alex Clark)
Date: Sun, 05 Feb 2012 13:48:17 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120204194054.GA29629@sleipnir.bytereef.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
Message-ID: <jgmiti$5p4$1@dough.gmane.org>

Hi Stefan,

On 2/4/12 2:40 PM, Stefan Krah wrote:
> Hello,
>
> I wonder what the point of a site like pythonpackages.com is. For my
> package (cdecimal)

I think the following explanation applies to all Python packages:

- http://pythonpackages.com/about#why

Download count is important, but it's certainly not the only criteria 
one can use to evaluate a package.

> it spreads misinformation on two counts:
>
>    1) The number of downloads cannot be established since cdecimal
>       is not hosted on PyPI.

pythonpackages.com considers this to be "zero downloads". if there was 
an API for off-site downloads to report back to PyPI then maybe we could 
use it to report those statistics on pythonpackages.com. AFAIK there is 
currently no such thing. (I used to have a tool tip in place that 
explained this; I'll put it back ASAP. crate.io has something similar in 
place IIRC.)


>
>    2) "Now supports Python 3!" is false since cdecimal has supported
>       Python3 from the very beginning.


Fair enough, I changed the message to "Supports Python 3". Is that 
better? FWIW there is an issue tracker here for this sort of thing:

- https://bitbucket.org/pythonpackages/pythonpackages.com/issues


>
>
> Otherwise it displays Google ads


 >>> True


> and the same information as PyPI,
> only in a less readable manner.

This is subjective, so I can't really offer an intelligent reply (though 
I can tell you that I know people typically either love or hate Twitter 
bootstrap; personally, I like it "enough".)


Alex




>
>
> Stefan Krah


-- 
Alex Clark ? http://pythonpackages.com


From ownerscircle at gmail.com  Mon Feb  6 01:01:55 2012
From: ownerscircle at gmail.com (PJ Eby)
Date: Sun, 5 Feb 2012 19:01:55 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgmiti$5p4$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgmiti$5p4$1@dough.gmane.org>
Message-ID: <CALeMXf4_jO4gFY1fM1fp5hAiUB5+Sm8JPcSS6Vs8UEmY4PuZLg@mail.gmail.com>

On Feb 5, 2012 1:48 PM, "Alex Clark" <aclark at aclark.net> wrote:
> pythonpackages.com considers this to be "zero downloads". if there was an
API for off-site downloads to report back to PyPI then maybe we could use
it to report those statistics on pythonpackages.com. AFAIK there is
currently no such thing. (I used to have a tool tip in place that explained
this; I'll put it back ASAP. crate.io has something similar in place IIRC.)

Perhaps instead of zero with a tooltip, you could just say "N/A" for
releases that don't list any files -- or just exclude the download line
altogether.

You do, after all, know whether a release has any files from its PyPI data,
so you do know the difference between "zero downloads" and "unknown
downloads".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120205/60bae36b/attachment.html>

From aclark at aclark.net  Mon Feb  6 04:22:50 2012
From: aclark at aclark.net (Alex Clark)
Date: Sun, 05 Feb 2012 22:22:50 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CALeMXf4_jO4gFY1fM1fp5hAiUB5+Sm8JPcSS6Vs8UEmY4PuZLg@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgmiti$5p4$1@dough.gmane.org>
	<CALeMXf4_jO4gFY1fM1fp5hAiUB5+Sm8JPcSS6Vs8UEmY4PuZLg@mail.gmail.com>
Message-ID: <jgnh2a$v79$1@dough.gmane.org>

On 2/5/12 7:01 PM, PJ Eby wrote:
>
> On Feb 5, 2012 1:48 PM, "Alex Clark" <aclark at aclark.net
> <mailto:aclark at aclark.net>> wrote:
>  > pythonpackages.com <http://pythonpackages.com> considers this to be
> "zero downloads". if there was an API for off-site downloads to report
> back to PyPI then maybe we could use it to report those statistics on
> pythonpackages.com <http://pythonpackages.com>. AFAIK there is currently
> no such thing. (I used to have a tool tip in place that explained this;
> I'll put it back ASAP. crate.io <http://crate.io> has something similar
> in place IIRC.)
>
> Perhaps instead of zero with a tooltip, you could just say "N/A" for
> releases that don't list any files -- or just exclude the download line
> altogether.

Done, e.g. http://pythonpackages.com/info/cdecimal


>
> You do, after all, know whether a release has any files from its PyPI
> data, so you do know the difference between "zero downloads" and
> "unknown downloads".


Well, it's a package that exists for which the API reports zero 
downloads. That means either there has been no release yet, or the 
package does not host its releases on PyPI, IIUC.


Alex


>
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig


-- 
Alex Clark ? http://pythonpackages.com


From faassen at startifact.com  Mon Feb  6 16:32:59 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Mon, 06 Feb 2012 16:32:59 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgmgrf$ndi$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
Message-ID: <jgorrb$c8m$1@dough.gmane.org>

On 02/05/2012 07:12 PM, Terry Reedy wrote:
> On 2/5/2012 8:37 AM, Kai Diefenbach wrote:
>> Why not? Packages, which are not hosted on PyPi suck.
>
> This is a technical discussion list, not a flame list.
> That comment is both wrong and unhelpful.

'suck' is not the right way to express the problem, and it's the 
original poster's choice to host somewhere else, but it can indeed be 
inconvenient to quite a few users of PyPI if a package is not hosted on 
PyPI.

This because setuptools (and thus, easy_install, pip, buildout) for 
better or for worse uses a "trawl the web" approach to find download 
links, and multiple sites to download from create multiple potential 
points of failure besides PyPI itself.

This makes setuptools work for a range of cases and that's nice, but 
it's also a drawback, because on a fairly regular basis I at least have 
had the issue that a package wasn't hosted on PyPI and that the site 
hosting the package was suddenly down or had changed, breaking the 
setuptools-based automatic download. If the package were hosted on PyPI 
I wouldn't have had this issue, as PyPI itself is actually tolerably 
reliable (especially with mirroring in place; but these external 
packages are also not mirrored).

Of course the response I'll undoubtedly get is that I should host these 
packages myself or include them in my version control system and all 
that. And yes, I can do this, and sometimes I do. But doing that is in 
this subjective user's opinion actually an inconvenience. Any 'pip' user 
that installs a package from PyPI that has dependencies listed in 
setup.py can run into this problem.

So the original poster could at least consider uploading their package 
on PyPI to lessen his complaint. Besides the web UI, they'll find handy 
tools available to help automate things, such as 'setup.py sdist upload' 
and for more power, zest.releaser. But of course they can choose not to 
do so at all too - that's the way things work [1].

Regards,

Martijn

[1] I suspect an alternate timeline in which setuptools had never done 
this web trawling and would only download from PyPI would have lead to a 
more pleasant situation for users, though I'm not sure: setuptools being 
able to download dependencies might've retarded adoption of setuptools 
instead.


From pydanny at gmail.com  Mon Feb  6 17:15:47 2012
From: pydanny at gmail.com (Daniel Greenfeld)
Date: Mon, 6 Feb 2012 08:15:47 -0800
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgorrb$c8m$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
Message-ID: <CAOoSJ_pMPLU8z5CMAsur=cY72TFYxYgsLV9kUrUL4NcFPa+JZA@mail.gmail.com>

On Mon, Feb 6, 2012 at 7:32 AM, Martijn Faassen <faassen at startifact.com> wrote:

> This because setuptools (and thus, easy_install, pip, buildout) for better
> or for worse uses a "trawl the web" approach to find download links, and
> multiple sites to download from create multiple potential points of failure
> besides PyPI itself.
>
> This makes setuptools work for a range of cases and that's nice, but it's
> also a drawback, because on a fairly regular basis I at least have had the
> issue that a package wasn't hosted on PyPI and that the site hosting the
> package was suddenly down or had changed, breaking the setuptools-based
> automatic download. If the package were hosted on PyPI I wouldn't have had
> this issue, as PyPI itself is actually tolerably reliable (especially with
> mirroring in place; but these external packages are also not mirrored).
>
> Of course the response I'll undoubtedly get is that I should host these
> packages myself or include them in my version control system and all that.
> And yes, I can do this, and sometimes I do. But doing that is in this
> subjective user's opinion actually an inconvenience. Any 'pip' user that
> installs a package from PyPI that has dependencies listed in setup.py can
> run into this problem.
>
> So the original poster could at least consider uploading their package on
> PyPI to lessen his complaint. Besides the web UI, they'll find handy tools
> available to help automate things, such as 'setup.py sdist upload' and for
> more power, zest.releaser. But of course they can choose not to do so at all
> too - that's the way things work [1].
>
> Regards,
>
> Martijn
>
> [1] I suspect an alternate timeline in which setuptools had never done this
> web trawling and would only download from PyPI would have lead to a more
> pleasant situation for users, though I'm not sure: setuptools being able to
> download dependencies might've retarded adoption of setuptools instead.

I agree 100% with Martijn. Maybe there was a time when hosting your
package off of PyPI was a good idea. I think if that time existed, it
has now passed. Normally I prefer giving package authors more control,
but this is one of those places where the users of the service ought
to be able to expect packages to all be found in one location.

-- 
'Knowledge is Power'
Daniel Greenfeld
http://pydanny.blogspot.com

From aclark at aclark.net  Mon Feb  6 18:13:29 2012
From: aclark at aclark.net (Alex Clark)
Date: Mon, 06 Feb 2012 12:13:29 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CAOoSJ_pMPLU8z5CMAsur=cY72TFYxYgsLV9kUrUL4NcFPa+JZA@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<CAOoSJ_pMPLU8z5CMAsur=cY72TFYxYgsLV9kUrUL4NcFPa+JZA@mail.gmail.com>
Message-ID: <jgp1np$vgi$1@dough.gmane.org>

Hi

On 2/6/12 11:15 AM, Daniel Greenfeld wrote:
> On Mon, Feb 6, 2012 at 7:32 AM, Martijn Faassen<faassen at startifact.com>  wrote:
>
>> This because setuptools (and thus, easy_install, pip, buildout) for better
>> or for worse uses a "trawl the web" approach to find download links, and
>> multiple sites to download from create multiple potential points of failure
>> besides PyPI itself.
>>
>> This makes setuptools work for a range of cases and that's nice, but it's
>> also a drawback, because on a fairly regular basis I at least have had the
>> issue that a package wasn't hosted on PyPI and that the site hosting the
>> package was suddenly down or had changed, breaking the setuptools-based
>> automatic download. If the package were hosted on PyPI I wouldn't have had
>> this issue, as PyPI itself is actually tolerably reliable (especially with
>> mirroring in place; but these external packages are also not mirrored).
>>
>> Of course the response I'll undoubtedly get is that I should host these
>> packages myself or include them in my version control system and all that.
>> And yes, I can do this, and sometimes I do. But doing that is in this
>> subjective user's opinion actually an inconvenience. Any 'pip' user that
>> installs a package from PyPI that has dependencies listed in setup.py can
>> run into this problem.
>>
>> So the original poster could at least consider uploading their package on
>> PyPI to lessen his complaint. Besides the web UI, they'll find handy tools
>> available to help automate things, such as 'setup.py sdist upload' and for
>> more power, zest.releaser. But of course they can choose not to do so at all
>> too - that's the way things work [1].
>>
>> Regards,
>>
>> Martijn
>>
>> [1] I suspect an alternate timeline in which setuptools had never done this
>> web trawling and would only download from PyPI would have lead to a more
>> pleasant situation for users, though I'm not sure: setuptools being able to
>> download dependencies might've retarded adoption of setuptools instead.
>
> I agree 100% with Martijn. Maybe there was a time when hosting your
> package off of PyPI was a good idea. I think if that time existed, it
> has now passed. Normally I prefer giving package authors more control,
> but this is one of those places where the users of the service ought
> to be able to expect packages to all be found in one location.


+1. And if you want to host your packages off-site I think that is 
perfectly reasonable. But it would make everyone's life easier if we 
knew that: for every release of a Python package on earth, there is a 
corresponding package on PyPI.

E.g. In Plone-land we currently encourage dual-releasing to both PyPI 
and plone.org/products. This has several benefits:

0. Easy content creation. Having nice product pages for our add-ons is a 
marketing win.

1. Everyone that runs buildout to install Plone can rely on packages 
being found on PyPI.

2. If PyPI goes down, those folks can use an official PyPI mirror to 
install the same set of packages[1]

3. If PyPI goes down, those folks can use plone.org/products[2] to 
install any packages released to plone.org/products.

There is also some disadvantage:

1. Folks rarely take advantage of #3. So I think in the future we may 
consider replacing plone.org/products with a full PyPI mirror and simply 
rely on mirroring instead of dual-releasing.

2. Folks sometimes don't dual-release. Implementing the change suggested 
in #1 of this list would fix that.


Alex


[1] In theory. I understand there has been some concern about the 
stability/integrity of some mirrors lately.


[2] http://dist.plone.org/packages/


>


-- 
Alex Clark ? http://pythonpackages.com


From aclark at aclark.net  Mon Feb  6 18:19:11 2012
From: aclark at aclark.net (Alex Clark)
Date: Mon, 06 Feb 2012 12:19:11 -0500
Subject: [Catalog-sig] Distutils sdist formats best practice
Message-ID: <jgp22g$2lq$1@dough.gmane.org>

Hi,

What is the best practice for creating sdists with varying compression 
formats?

I just noticed that the format is largely platform-specific[1]. So I'm 
inclined to have pythonpackages.com create all popular formats and make 
them available for download, i.e.:

- bzip2
- zip
- gzip


What do pip/easy_install/etc do when they encounter both a .zip and a 
.tar.gz, for example? Is it "better" or "worse" to upload more than a 
single format?


Alex



[1] http://docs.python.org/distutils/sourcedist.html



-- 
Alex Clark ? http://pythonpackages.com


From hanno at hannosch.eu  Mon Feb  6 18:26:21 2012
From: hanno at hannosch.eu (Hanno Schlichting)
Date: Mon, 6 Feb 2012 18:26:21 +0100
Subject: [Catalog-sig] Distutils sdist formats best practice
In-Reply-To: <jgp22g$2lq$1@dough.gmane.org>
References: <jgp22g$2lq$1@dough.gmane.org>
Message-ID: <CAJ5sox5pcxbXoGSDGuDN4nqHwu38UuGjSNPmJxLnDS0GZdJ88g@mail.gmail.com>

On Mon, Feb 6, 2012 at 6:19 PM, Alex Clark <aclark at aclark.net> wrote:
> What is the best practice for creating sdists with varying compression
> formats?

In the past it was best to create .zip files for all platforms. This
was largely to avoid problems in Python's tarfile module in Python 2.4
and maybe 2.5, which failed to extract files whose combined path and
file name was exactly 100 characters long.

These days that's likely no longer a major concern.

Personally I'd still use zip files for all platforms, as all of them
by default have tools to open and extract these files. On *nix
platforms it's also not always clear if Python was built with bzip2
support or if that option was omitted.

Hanno

From donald.stufft at gmail.com  Mon Feb  6 18:37:41 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Mon, 6 Feb 2012 12:37:41 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgp1np$vgi$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<CAOoSJ_pMPLU8z5CMAsur=cY72TFYxYgsLV9kUrUL4NcFPa+JZA@mail.gmail.com>
	<jgp1np$vgi$1@dough.gmane.org>
Message-ID: <A72C703D4E144F0FA3DC662B378C3DE7@gmail.com>

A big +1 to hosting everything on PyPI  


On Monday, February 6, 2012 at 12:13 PM, Alex Clark wrote:

> Hi
>  
> On 2/6/12 11:15 AM, Daniel Greenfeld wrote:
> > On Mon, Feb 6, 2012 at 7:32 AM, Martijn Faassen<faassen at startifact.com (mailto:faassen at startifact.com)> wrote:
> >  
> > > This because setuptools (and thus, easy_install, pip, buildout) for better
> > > or for worse uses a "trawl the web" approach to find download links, and
> > > multiple sites to download from create multiple potential points of failure
> > > besides PyPI itself.
> > >  
> > > This makes setuptools work for a range of cases and that's nice, but it's
> > > also a drawback, because on a fairly regular basis I at least have had the
> > > issue that a package wasn't hosted on PyPI and that the site hosting the
> > > package was suddenly down or had changed, breaking the setuptools-based
> > > automatic download. If the package were hosted on PyPI I wouldn't have had
> > > this issue, as PyPI itself is actually tolerably reliable (especially with
> > > mirroring in place; but these external packages are also not mirrored).
> > >  
> > > Of course the response I'll undoubtedly get is that I should host these
> > > packages myself or include them in my version control system and all that.
> > > And yes, I can do this, and sometimes I do. But doing that is in this
> > > subjective user's opinion actually an inconvenience. Any 'pip' user that
> > > installs a package from PyPI that has dependencies listed in setup.py can
> > > run into this problem.
> > >  
> > > So the original poster could at least consider uploading their package on
> > > PyPI to lessen his complaint. Besides the web UI, they'll find handy tools
> > > available to help automate things, such as 'setup.py sdist upload' and for
> > > more power, zest.releaser. But of course they can choose not to do so at all
> > > too - that's the way things work [1].
> > >  
> > > Regards,
> > >  
> > > Martijn
> > >  
> > > [1] I suspect an alternate timeline in which setuptools had never done this
> > > web trawling and would only download from PyPI would have lead to a more
> > > pleasant situation for users, though I'm not sure: setuptools being able to
> > > download dependencies might've retarded adoption of setuptools instead.
> > >  
> >  
> >  
> > I agree 100% with Martijn. Maybe there was a time when hosting your
> > package off of PyPI was a good idea. I think if that time existed, it
> > has now passed. Normally I prefer giving package authors more control,
> > but this is one of those places where the users of the service ought
> > to be able to expect packages to all be found in one location.
> >  
>  
>  
>  
> +1. And if you want to host your packages off-site I think that is  
> perfectly reasonable. But it would make everyone's life easier if we  
> knew that: for every release of a Python package on earth, there is a  
> corresponding package on PyPI.
>  
> E.g. In Plone-land we currently encourage dual-releasing to both PyPI  
> and plone.org/products (http://plone.org/products). This has several benefits:
>  
> 0. Easy content creation. Having nice product pages for our add-ons is a  
> marketing win.
>  
> 1. Everyone that runs buildout to install Plone can rely on packages  
> being found on PyPI.
>  
> 2. If PyPI goes down, those folks can use an official PyPI mirror to  
> install the same set of packages[1]
>  
> 3. If PyPI goes down, those folks can use plone.org/products[2] (http://plone.org/products[2]) to  
> install any packages released to plone.org/products (http://plone.org/products).
>  
> There is also some disadvantage:
>  
> 1. Folks rarely take advantage of #3. So I think in the future we may  
> consider replacing plone.org/products (http://plone.org/products) with a full PyPI mirror and simply  
> rely on mirroring instead of dual-releasing.
>  
> 2. Folks sometimes don't dual-release. Implementing the change suggested  
> in #1 of this list would fix that.
>  
>  
> Alex
>  
>  
> [1] In theory. I understand there has been some concern about the  
> stability/integrity of some mirrors lately.
>  
>  
> [2] http://dist.plone.org/packages/
>  
>  
>  
>  
> --  
> Alex Clark ? http://pythonpackages.com
>  
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
>  
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120206/7cecf008/attachment.html>

From stefan-usenet at bytereef.org  Mon Feb  6 21:08:23 2012
From: stefan-usenet at bytereef.org (Stefan Krah)
Date: Mon, 6 Feb 2012 21:08:23 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgorrb$c8m$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
Message-ID: <20120206200823.GA8910@sleipnir.bytereef.org>

Martijn Faassen <faassen at startifact.com> wrote:
> original poster's choice to host somewhere else, but it can indeed be  
> inconvenient to quite a few users of PyPI if a package is not hosted on  
> PyPI.

I don't see any inconvenience since bytereef.org has a comparable
uptime to python.org.

I've listed my reasons for not hosting on PyPI earlier here:

http://mail.python.org/pipermail/catalog-sig/2011-May/003746.html


Stefan Krah



From stefan-usenet at bytereef.org  Mon Feb  6 21:14:31 2012
From: stefan-usenet at bytereef.org (Stefan Krah)
Date: Mon, 6 Feb 2012 21:14:31 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgnh2a$v79$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgmiti$5p4$1@dough.gmane.org>
	<CALeMXf4_jO4gFY1fM1fp5hAiUB5+Sm8JPcSS6Vs8UEmY4PuZLg@mail.gmail.com>
	<jgnh2a$v79$1@dough.gmane.org>
Message-ID: <20120206201431.GA9005@sleipnir.bytereef.org>

Alex Clark <aclark at aclark.net> wrote:
>> Perhaps instead of zero with a tooltip, you could just say "N/A" for
>> releases that don't list any files -- or just exclude the download line
>> altogether.
>
> Done, e.g. http://pythonpackages.com/info/cdecimal

Thanks, that is more informative.


Stefan Krah



From lists at zopyx.com  Mon Feb  6 21:17:37 2012
From: lists at zopyx.com (Andreas Jung)
Date: Mon, 06 Feb 2012 21:17:37 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120206200823.GA8910@sleipnir.bytereef.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
Message-ID: <4F303561.6010404@zopyx.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Stefan Krah wrote:
> Martijn Faassen <faassen at startifact.com> wrote:
>> original poster's choice to host somewhere else, but it can indeed
>> be inconvenient to quite a few users of PyPI if a package is not
>> hosted on PyPI.
> 
> I don't see any inconvenience since bytereef.org has a comparable 
> uptime to python.org.

Not an argument. It is in the interest of all serious Python developers
that Python packages are maintained in a proper way on PyPI
(documentation, hosting, metadata etc.). Having a package on a private
server is often a single-point-of-failure and not acceptable for
professional deployments. My point about this: if a person does not want
to host its package on PyPi than it should stay away from PyPI. Package
hygiene and a certain level of professional package repository is more
important and personal reasons for not hosting packages on PyPI.

- -aj
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJPMDVgAAoJEADcfz7u4AZjgLULv0CxGU34t6C3N/XYW9U4OS1r
tNWH5m5VEFJ8gqOkXrnzK2XldyNt4y1LeOAVoYtciuJt9PB02zBFobbGwS7OoKcZ
V7cu23LQeo3RQG2NeQ5UTDcEBKioHDWTKayPFRokVcbwo+DRbPQzph7Q+51bfugz
kcH248laGq9zx5W3tsuH0dDYPh/p5HQPwzGk3mM0dlqZi6Ztshe+DE1xLc+0Su3D
Psx/4Ax7CYCUJUKUUT3ondzDdThmjq8iT5gCWhGDEAm6qEpt+EgP1dqT6Lv0tzek
9e2FM4d2aHhdxYfe9RfWkmu61PT42cCfvNSmihVFsCBbqQMEBKejdIbDNwNjWl9k
D6Y4SOmJYpW0LL07IEHNmbj5ARaXUg+0fXoIgw4BDksu7WfDH7MZREEzm4Wqaaze
ILHaLSyNAd1nivHe4q6QXe9Y8ZhAehTTPjCzCvZmcgQuRbu1Apxw4RKWSYfMGIFZ
zjwEOrcd2rWPqu6CZ/zjAQoFefTACxs=
=3DWe
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120206/5bf19b67/attachment.vcf>

From donald.stufft at gmail.com  Mon Feb  6 21:25:38 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Mon, 6 Feb 2012 15:25:38 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120206200823.GA8910@sleipnir.bytereef.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
Message-ID: <4A3563ECDA99463F80770E786788EAA7@gmail.com>

It's your prerogative to host it where you will, but just from my personal point of view:


On Monday, February 6, 2012 at 3:08 PM, Stefan Krah wrote:

> Martijn Faassen <faassen at startifact.com (mailto:faassen at startifact.com)> wrote:
> > original poster's choice to host somewhere else, but it can indeed be 
> > inconvenient to quite a few users of PyPI if a package is not hosted on 
> > PyPI.
> > 
> 
> 
> I don't see any inconvenience since bytereef.org (http://bytereef.org) has a comparable
> uptime to python.org (http://python.org).
> 
> 

Even if your server has a _better_ uptime than PyPI, the combined downtime will be worse. (If PyPI is down the
user cannot install your package, or any package, if your server is down, but PyPI is up they cannot install your
package but can other packages.) 
> 
> I've listed my reasons for not hosting on PyPI earlier here:
> 
> http://mail.python.org/pipermail/catalog-sig/2011-May/003746.html
> 
> 
> Stefan Krah
To address what was said here:
 
1) This is a valid complaint and should be brought up as a reason to amend the ToS (not particularly sure what the ToS is for PyPI tbh)

2) This complaint isn't particularly valid in the sense of should I upload my files to PyPI. You are already uploading the metadata that
they are scraping, otherwise you would be unable to list on PyPI. Wether or not you have a file hosted there won't make one bit of 
difference to Google.

3) This is valid as well, I would argue that you should push for better package download stats for the authors of packages.

4) I think this is an invalid point as well, it's quite easy to add a Home Page metadata (as you already have done), and to
make your long_description state that the primary page for your package is at http://www.bytereef.org/mpdecimal/index.html
and to go there for that information.

5) I addressed this above but i'll reiterate this point, unless your server has (actual, not theoretical) 100% uptime as well as all
the networking routes between it and the end user, people installing your package have a greater chance of being unable to
install your package than if you just hosted on PyPi. You cannot remove their dependency on PyPI being up, but adding another
possible place for failure means that the combined uptime of your package being installable is lower.

Theoertical 99% uptime of PyPI and 99% uptime of your server would mean a combined 98% uptime. And that's without getting
into the additional points of failure between the person wanting to use your package and your server.
> 
> 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120206/97fc01fe/attachment-0001.html>

From faassen at startifact.com  Mon Feb  6 21:33:27 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Mon, 06 Feb 2012 21:33:27 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120206200823.GA8910@sleipnir.bytereef.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
Message-ID: <jgpden$3aq$1@dough.gmane.org>

On 02/06/2012 09:08 PM, Stefan Krah wrote:
> Martijn Faassen<faassen at startifact.com>  wrote:
>> original poster's choice to host somewhere else, but it can indeed be
>> inconvenient to quite a few users of PyPI if a package is not hosted on
>> PyPI.
>
> I don't see any inconvenience since bytereef.org has a comparable
> uptime to python.org.

I've experienced a site which was hosting a Python package which had 
awesome uptime, but then something was screwed up about the security of 
the host at some point and while it remained up, it took forever 
(months? years?) to get resolved.

So anyway, it's great bytereef.org has great uptime, but it's also clear 
relying on 10 sites, even if each have a great uptime and network 
reachability is going to give me worse uptime than having to rely on 
one, if that one has a reasonable uptime. Unless of course the content 
is mirrored, in which case reliability goes up.

> I've listed my reasons for not hosting on PyPI earlier here:
>
> http://mail.python.org/pipermail/catalog-sig/2011-May/003746.html

Interesting, and thank you for the reference. Your reasons make sense of 
course, though they don't tip the balance for me personally. I notice 
all your reasons (besides the uptime one) are about control over your 
package as the author. The arguments I've made from the perspective of 
package users. I'm a package author too; plenty of my stuff on PyPI, but 
I'm an even bigger user of other people's code.

Regards,

Martijn


From stefan-usenet at bytereef.org  Mon Feb  6 21:55:37 2012
From: stefan-usenet at bytereef.org (Stefan Krah)
Date: Mon, 6 Feb 2012 21:55:37 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <4F303561.6010404@zopyx.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
Message-ID: <20120206205537.GA9149@sleipnir.bytereef.org>

Andreas Jung <lists at zopyx.com> wrote:
> > I don't see any inconvenience since bytereef.org has a comparable 
> > uptime to python.org.
> 
> Not an argument. It is in the interest of all serious Python developers
> that Python packages are maintained in a proper way on PyPI
> (documentation, hosting, metadata etc.). Having a package on a private
> server is often a single-point-of-failure and not acceptable for
> professional deployments.

Martijn Faassen has predicted that this would come up, so here it goes:

If that's a point of failure then you are simply not doing a professional
deployment.

People who need guarantees like that should maintain their own package
repository or try Enthought, ActiveState, etc.



Stefan Krah



From stefan-usenet at bytereef.org  Mon Feb  6 22:35:27 2012
From: stefan-usenet at bytereef.org (Stefan Krah)
Date: Mon, 6 Feb 2012 22:35:27 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgpden$3aq$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<jgpden$3aq$1@dough.gmane.org>
Message-ID: <20120206213527.GA9285@sleipnir.bytereef.org>

Martijn Faassen <faassen at startifact.com> wrote:
> On 02/06/2012 09:08 PM, Stefan Krah wrote:
>> I don't see any inconvenience since bytereef.org has a comparable
>> uptime to python.org.
>
> I've experienced a site which was hosting a Python package which had  
> awesome uptime, but then something was screwed up about the security of  
> the host at some point and while it remained up, it took forever  
> (months? years?) to get resolved.

And? I'm not exactly unreachable and I doubt there will be a security problem.
Furthermore I'm posting the sha256sums of the packages in the announcements,
so they are archived on several mailing lists.

For the general case I'd suggest that PyPI gives an author the option to
tie an sha256sum to a package version *once*. This leaves an opportunity
to correct a release (recent discussion), but as soon as the checksum is
published it cannot be altered.

If a package is removed entirely, any version numbers that have been used
would need to be stored intenally to prevent a re-upload with the same name
but a different checksum.


The download tools would need to get the capability to verify the checksum.


Stefan Krah



From pydanny at gmail.com  Mon Feb  6 22:42:02 2012
From: pydanny at gmail.com (Daniel Greenfeld)
Date: Mon, 6 Feb 2012 13:42:02 -0800
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120206205537.GA9149@sleipnir.bytereef.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
	<20120206205537.GA9149@sleipnir.bytereef.org>
Message-ID: <CAOoSJ_o3EFuFsSj1V-ucJbyxYZOTxyYVLUFaC=oe7wk4w6iG5w@mail.gmail.com>

On Mon, Feb 6, 2012 at 12:55 PM, Stefan Krah <stefan-usenet at bytereef.org> wrote:
> Andreas Jung <lists at zopyx.com> wrote:
>> > I don't see any inconvenience since bytereef.org has a comparable
>> > uptime to python.org.
>>
>> Not an argument. It is in the interest of all serious Python developers
>> that Python packages are maintained in a proper way on PyPI
>> (documentation, hosting, metadata etc.). Having a package on a private
>> server is often a single-point-of-failure and not acceptable for
>> professional deployments.
>
> Martijn Faassen has predicted that this would come up, so here it goes:
>
> If that's a point of failure then you are simply not doing a professional
> deployment.
>
> People who need guarantees like that should maintain their own package
> repository or try Enthought, ActiveState, etc.

I've been told that 'professional deployments should not been done
from PyPI for years. That's always irked me quite a bit, and I think
that things should change. In fact, I contend that as PyPI is the
canonical place for package listings, then that sentence is incredible
dismaying/shocking for new users of Python. How do Fedora/Ubuntu/Perl
and other systems work? Are their systems also modeled the same way?

So why can't PyPI become the best place for package downloads? The
technical obstacles are being overcome with mirrors and improved
architecture. As that occurs, I think a requirement for PyPI listing
should be that a copy of the posted version is on the site.

-- 
'Knowledge is Power'
Daniel Greenfeld
http://pydanny.blogspot.com

From tjreedy at udel.edu  Mon Feb  6 22:58:11 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 06 Feb 2012 16:58:11 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <4F303561.6010404@zopyx.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
Message-ID: <jgpie0$a8s$1@dough.gmane.org>

On 2/6/2012 3:17 PM, Andreas Jung wrote:
> My point about this: if a person does not want
> to host its package on PyPi than it should stay away from PyPI.

The Python Package Index was originally just that: a package *INDEX*, 
aiming to be a complete index. It did not originate the idea of such an 
index, but has pretty much superseded previous 'unofficial' efforts.

Now you want to censor it to meet *your* needs, to only list packages 
that *you* are interested in.

If I remember correctly, the Cheeseshop/PyPI was originally *just* an 
index. The hosting-repository service was added later -- as a 
convenience firstly to authors. I now believe that the repository should 
have been and should be kept separate, as the Python Package Repository 
-- PyPaR. Then repository issues would be clearly separate from index 
issues.

-- 
Terry Jan Reedy


From jacob at jacobian.org  Mon Feb  6 23:11:56 2012
From: jacob at jacobian.org (Jacob Kaplan-Moss)
Date: Mon, 06 Feb 2012 16:11:56 -0600
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CAOoSJ_o3EFuFsSj1V-ucJbyxYZOTxyYVLUFaC=oe7wk4w6iG5w@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
	<20120206205537.GA9149@sleipnir.bytereef.org>
	<CAOoSJ_o3EFuFsSj1V-ucJbyxYZOTxyYVLUFaC=oe7wk4w6iG5w@mail.gmail.com>
Message-ID: <4F30502C.3060608@jacobian.org>

Daniel Greenfeld wrote:
> I've been told that 'professional deployments should not been done
> from PyPI for years. That's always irked me quite a bit, and I think
> that things should change.

I completely agree.

I'm one of those people who's told you that you can't reply on PyPI for
repeatable, bullet-proof deployments. That's a statement of fact, but
it's not a situation I'm particularly happy with. It'd be a pretty great
thing for our community if we could improve PyPI to the point that it
has sufficient reliability.

It's not a particularly hard problem technically, but it is going to
require us to stop being content with the status quo and push PyPI forward.

Alex, Donald, et al. -- keep up the good work!

Jacob

From tjreedy at udel.edu  Mon Feb  6 23:14:36 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 06 Feb 2012 17:14:36 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <4A3563ECDA99463F80770E786788EAA7@gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4A3563ECDA99463F80770E786788EAA7@gmail.com>
Message-ID: <jgpjcp$hgj$1@dough.gmane.org>

On 2/6/2012 3:25 PM, Donald Stufft wrote:

>> I've listed my reasons for not hosting on PyPI earlier here:
>>
>> http://mail.python.org/pipermail/catalog-sig/2011-May/003746.html
>>
>>
>> Stefan Krah
> To address what was said here:
> 1) This is a valid complaint and should be brought up as a reason to
> amend the ToS (not particularly sure what the ToS is for PyPI tbh)

The previous discussion of the license and associated terms, after the 
license was unilaterally revised, made it clear that hosting was a 
service to authors made available on a take-it-or-leave-it basis with no 
discussion wanted or considered. I remember the message as "If you do 
not accept the new license, remove existing uploads and do not upload in 
the future."

-- 
Terry Jan Reedy


From faassen at startifact.com  Tue Feb  7 01:41:14 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Tue, 07 Feb 2012 01:41:14 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120206213527.GA9285@sleipnir.bytereef.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<jgpden$3aq$1@dough.gmane.org>
	<20120206213527.GA9285@sleipnir.bytereef.org>
Message-ID: <jgprva$cq5$1@dough.gmane.org>

On 02/06/2012 10:35 PM, Stefan Krah wrote:
> Martijn Faassen<faassen at startifact.com>  wrote:
>> On 02/06/2012 09:08 PM, Stefan Krah wrote:
>>> I don't see any inconvenience since bytereef.org has a comparable
>>> uptime to python.org.
>>
>> I've experienced a site which was hosting a Python package which had
>> awesome uptime, but then something was screwed up about the security of
>> the host at some point and while it remained up, it took forever
>> (months? years?) to get resolved.
>
> And? I'm not exactly unreachable and I doubt there will be a security problem.
> Furthermore I'm posting the sha256sums of the packages in the announcements,
> so they are archived on several mailing lists.

Taking you out of the picture, if there are 2 sites that I need to rely 
on, both with equally great uptime and security and reachability, the 
chances of problems at any given time is higher than if I just had to 
rely on 1 such site.

Multiple sites can only increase reliability if they both provide the 
same services.

I'm not telling you that you shouldn't be hosting your stuff. I'm saying 
that in general people hosting their own stuff, while entirely within 
their rights, is less great for users.

> For the general case I'd suggest that PyPI gives an author the option to
> tie an sha256sum to a package version *once*. This leaves an opportunity
> to correct a release (recent discussion), but as soon as the checksum is
> published it cannot be altered.

That's an interesting idea!

> If a package is removed entirely, any version numbers that have been used
> would need to be stored intenally to prevent a re-upload with the same name
> but a different checksum.
>
> The download tools would need to get the capability to verify the checksum.

I agree, and the upload tools would need support for this too.

Regards,

Martijn



From faassen at startifact.com  Tue Feb  7 02:17:57 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Tue, 07 Feb 2012 02:17:57 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgpie0$a8s$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
Message-ID: <jgpu45$q41$1@dough.gmane.org>

On 02/06/2012 10:58 PM, Terry Reedy wrote:

[snip stuff about what people ought to do or be left free not to do. I 
think this list has too many discussions on morality]

> If I remember correctly, the Cheeseshop/PyPI was originally *just* an
> index.

I remember endless "Python needs a CPAN" discussions way back when, with 
people saying they think Python's great but CPAN is what makes Perl 
awesome. So I think that package hosting was always also on people's minds.

It had trouble getting off the ground as the task seemed daunting, so 
some very clever people decided to work on distutils and metadata first 
and then indexing metadata, then uploading packages, then automatic 
download facilities, etc, going for a CPAN-style network step by step, 
where different steps could be accomplished by a different set of people 
if necessary.

PEP 301 gives some historical background about the thinking in the 
rationale, which I suspect wasn't updated a lot since first publication 
so hopefully reflects the actual thinking of the time. It confirms that 
*this* particular step deliberately didn't think about repository 
functionality yet but it's also clear it's on people's minds as well as 
a future step:

http://www.python.org/dev/peps/pep-0301/

Already in 2001 PEP 241 was mentioning the sdist command, so that has 
been around for a while.

'sdist upload' and repository functionality was added in Python 2.5, so 
in 2006:

http://docs.python.org/release/2.5/whatsnew/pep-314.html.

But setuptools was also around at that time and was also supporting the 
'upload' command, including for python 2.3. Ah, I see a checkin here 
that seems to indicate it was added in the summer of 2005, in version 0.5a7:

http://www.gossamer-threads.com/lists/python/checkins/413647?do=post_view_threaded

so repository support in PyPI must be at least as old as 2005.

A few years ago I wrote a history about this stuff:

http://faassen.n--tree.net/blog/view/weblog/2009/11/09/0

Anyway, whatever the original purpose was, it's less relevant today, as 
it's more interesting what the purpose to authors and users the system 
has today. But I just like puzzling together the history so that's why 
the little side trip.

> The hosting-repository service was added later -- as a
> convenience firstly to authors. I now believe that the repository should
> have been and should be kept separate, as the Python Package Repository
> -- PyPaR. Then repository issues would be clearly separate from index
> issues.

Point taken about the separation of concerns.

Anyway, if that had happened, I think you would get people recommending 
that people put stuff both in the index and in the repository for the 
convenience of the users. And authors might want an integrated UI to 
manage things too. :)

I suspect in this alternate timeline it would've been easier to get 
people behind the notion of package hosting by a repository if it were 
construed as a *cache* of releases that are listed in the index.  People 
might feel different about the situation in such a case.

Regards,

Martijn


From richard at python.org  Tue Feb  7 02:40:33 2012
From: richard at python.org (Richard Jones)
Date: Tue, 7 Feb 2012 12:40:33 +1100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgpu45$q41$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgpu45$q41$1@dough.gmane.org>
Message-ID: <CAHrZfZBcKr5f7=HWxOnNr9ynqBfOP-AH=oKafSBvdk+QE_VUuA@mail.gmail.com>

On 7 February 2012 12:17, Martijn Faassen <faassen at startifact.com> wrote:
> so repository support in PyPI must be at least as old as 2005.

It was added during the sprints at US PyCon 2005 in DC. Here's a
horribly blurry photo (taken with a phone camera before such things
were worth a damn) of the work being done in part by Martin (who
implemented the GPG signature side of the upload - IIRC Mick was
working on the new template-based rendering code while I was working
on the actual upload command):

http://www.flickr.com/photos/richard_jones/7024147/in/set-72157594192581620/

I had always intended for PyPI to be a repository like CPAN so that
package authors could have a convenient place to distribute their
files. Just like we now have a convenient place for them to also host
documentation (though RTFD now also provides hosting for this.)


     Richard

From tjreedy at udel.edu  Tue Feb  7 05:09:45 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 06 Feb 2012 23:09:45 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgpu45$q41$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgpu45$q41$1@dough.gmane.org>
Message-ID: <jgq86m$l55$1@dough.gmane.org>

On 2/6/2012 8:17 PM, Martijn Faassen wrote:
>> The hosting-repository service was added later -- as a
>> convenience firstly to authors. I now believe that the repository should
>> have been and should be kept separate, as the Python Package Repository
>> -- PyPaR. Then repository issues would be clearly separate from index
>> issues.
...
> Anyway, if that had happened,

And I think it still should...

 > I think you would get people recommending
> that people put stuff both in the index and in the repository for the
> convenience of the users. And authors might want an integrated UI to
> manage things too. :)

Of course, uploading would mean being part of the index, just as today. 
But I would hope that such a separation would avert the calls to 
restrict the index. I think that would be destructive. If it were to 
occur, surely someone else would set a new index to Python packages that 
is at least as inclusive as PyPI is today.

Oh, and about PyPI download statistics, the original subject of this 
thread:
http://pypi.python.org/pypi/numpy/1.6.1
says numpy1.6.1 has 50000 downloads since last July.
http://sourceforge.net/projects/numpy/?source=directory
says 9000 just last week. So counting only PyPI downloads gives an 
undercount by a factor of perhaps 5.

Or http://pypi.python.org/pypi/MySQL-python/1.2.3
275K downloads since 2010 July versus
http://sourceforge.net/directory/os:windows/?q=python
http://sourceforge.net/projects/mysql-python/?source=directory
115K downloads per week (down to 96K last week)

-- 
Terry Jan Reedy


From lists at zopyx.com  Tue Feb  7 06:28:38 2012
From: lists at zopyx.com (Andreas Jung)
Date: Tue, 07 Feb 2012 06:28:38 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgpie0$a8s$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
Message-ID: <4F30B686.6040101@zopyx.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Terry Reedy wrote:
> On 2/6/2012 3:17 PM, Andreas Jung wrote:
>> My point about this: if a person does not want to host its package
>> on PyPi than it should stay away from PyPI.
> 
> The Python Package Index was originally just that: a package
> *INDEX*, aiming to be a complete index. It did not originate the idea
> of such an index, but has pretty much superseded previous
> 'unofficial' efforts.
> 
> Now you want to censor it to meet *your* needs, to only list
> packages that *you* are interested in.
> 
> If I remember correctly, the Cheeseshop/PyPI was originally *just*
> an index. The hosting-repository service was added later -- as a 
> convenience firstly to authors. I now believe that the repository
> should have been and should be kept separate, as the Python Package
> Repository -- PyPaR. Then repository issues would be clearly separate
> from index issues.

I don't care how something was originally used. I care about how things
are actually used and especially how things are used in a professional
environment. I am not in the business of tinkering and enterprises I
worded for or projects I was involved in require a *professsional*
and *reliable* Python packaging system. *I* as Python developer
know the issues and can deal with the current state. *Lots* of
internal developers that are using Python just a tool among others
often complain about the situtation and can not deal or won't
the complex aspects like setup a local package package or repository.
They expect things to just work.

Honestly I am truly pissed of the by arrogance and ignorance of package
maintainers coming with the very same arguments every time for *not
hosting* at least copies on PyPI. So my clear message is: if you don't
care about the professional developers and theirs by not hosting
packages on PyPI then please stay away...

- -aj

> 

- -- 
ZOPYX Limited           | zopyx group
Charlottenstr. 37/1     | The full-service network for Zope & Plone
D-72070 T?bingen        | Produce & Publish
www.zopyx.com           | www.produce-and-publish.com
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJPMLaGAAoJEADcfz7u4AZjRfgLv0M0MLId0kqO6qcSyA2vOy7D
4MV/c1tzo7d4fUDse8o5/DM+2ttOcswk0oPe4CaC1PShHxwaznyGkzGNjApWqPt8
W40/kOyE0HdBpUAvnzSWOV3Ujss6EkDDyuygV8mysl/KSwnq6X/lCnRQHX+Eb14/
VxbJIWc6ybSiTnOfojMnTQuSrz8M6Ko0qO/eR+7oGewQCS7jCADQH8PMWOuWgG03
ncrVwHWtqf5PSYFU3Bfg+pReub8bHYKChcYqtohXfGwSm7yxlccKDe5cKLG8pi5v
3pfyRLv6D4grzr2hTvoYnasekbU3WtWTvtL9Bzj6cens4GdXxcNLlydUvZd86IMY
0hfzl0Qxhz6kmYbnmRRbPuJijG4OxAHdko9I0IpgM3PKf4Xivxy4xrO71SShcO43
WWRce1MGV5fc1tuzoAp/lszfD8dx/tkvp1EPQSbyvjZzEZl48c2lI0BpahXual3m
l9nzWjAZaObjxzgXthyco2YcKXwJm6Q=
=7i3L
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120207/d93ca4f3/attachment-0001.vcf>

From kai.diefenbach at iqpp.de  Tue Feb  7 07:18:19 2012
From: kai.diefenbach at iqpp.de (Kai Diefenbach)
Date: Tue, 7 Feb 2012 07:18:19 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
Message-ID: <jgqfnd$vvj$1@dough.gmane.org>

On 2012-02-06 21:58:11 +0000, Terry Reedy said:

> On 2/6/2012 3:17 PM, Andreas Jung wrote:
>> My point about this: if a person does not want
>> to host its package on PyPi than it should stay away from PyPI.
> 
> The Python Package Index was originally just that: a package *INDEX*,
> aiming to be a complete index.

If a listed package is not available (because an external server is 
down) the index is broken.

Kai



From stefan-usenet at bytereef.org  Tue Feb  7 11:16:09 2012
From: stefan-usenet at bytereef.org (Stefan Krah)
Date: Tue, 7 Feb 2012 11:16:09 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <4F30B686.6040101@zopyx.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<4F30B686.6040101@zopyx.com>
Message-ID: <20120207101609.GA12442@sleipnir.bytereef.org>

Andreas Jung <lists at zopyx.com> wrote:
> Honestly I am truly pissed of the by arrogance and ignorance of package
> maintainers coming with the very same arguments every time for *not
> hosting* at least copies on PyPI. So my clear message is: if you don't
> care about the professional developers and theirs by not hosting
> packages on PyPI then please stay away...

While you were busy listing your demands, in the last 24 hours three major
international banks have successfully downloaded the cdecimal package.

My target audience is well aware of best practices. In fact, I provide
greater security than PyPI by publishing sha256sums on the announce list
when a package is released.


What you call "professional development" is just a euphemism for convenience
coupled with a false sense of security. No one can guarantee sanity for each
of the 18000+ packages.

Downloading is not the bottleneck, briefly auditing and making sure that
a package actually installs is. Python 3 compatibility is another *real*
issue, so perhaps you might want to upgrade your own packages.


Stefan Krah



From mal at egenix.com  Tue Feb  7 11:22:59 2012
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 07 Feb 2012 11:22:59 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120207101609.GA12442@sleipnir.bytereef.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<4F30B686.6040101@zopyx.com>
	<20120207101609.GA12442@sleipnir.bytereef.org>
Message-ID: <4F30FB83.3080706@egenix.com>

Guys, can we please stop this flame war and agree to disagree ?!

The current discussion is not leading anywhere.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 07 2012)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From lists at zopyx.com  Tue Feb  7 11:41:36 2012
From: lists at zopyx.com (Andreas Jung)
Date: Tue, 07 Feb 2012 11:41:36 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120207101609.GA12442@sleipnir.bytereef.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<4F30B686.6040101@zopyx.com>
	<20120207101609.GA12442@sleipnir.bytereef.org>
Message-ID: <4F30FFE0.60404@zopyx.com>

Don't feel offended..I don't care about your module. I am speaking of the
general situation.

-aj

Stefan Krah wrote:
> Andreas Jung <lists at zopyx.com> wrote:
>> Honestly I am truly pissed of the by arrogance and ignorance of package
>> maintainers coming with the very same arguments every time for *not
>> hosting* at least copies on PyPI. So my clear message is: if you don't
>> care about the professional developers and theirs by not hosting
>> packages on PyPI then please stay away...
>
> While you were busy listing your demands, in the last 24 hours three major
> international banks have successfully downloaded the cdecimal package.
>
> My target audience is well aware of best practices. In fact, I provide
> greater security than PyPI by publishing sha256sums on the announce list
> when a package is released.
>
>
> What you call "professional development" is just a euphemism for convenience
> coupled with a false sense of security. No one can guarantee sanity for each
> of the 18000+ packages.
>
> Downloading is not the bottleneck, briefly auditing and making sure that
> a package actually installs is. Python 3 compatibility is another *real*
> issue, so perhaps you might want to upgrade your own packages.
>
>
> Stefan Krah
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 324 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120207/817e1642/attachment.vcf>

From faassen at startifact.com  Tue Feb  7 17:01:30 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Tue, 7 Feb 2012 17:01:30 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CAHrZfZBcKr5f7=HWxOnNr9ynqBfOP-AH=oKafSBvdk+QE_VUuA@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgpu45$q41$1@dough.gmane.org>
	<CAHrZfZBcKr5f7=HWxOnNr9ynqBfOP-AH=oKafSBvdk+QE_VUuA@mail.gmail.com>
Message-ID: <CAGT4ZFheNMPqn5wf7BT-PKQvb0Ae4ek1emhhr+hoVYe-ReQKDw@mail.gmail.com>

Hey,

Cool, thanks for that bit! It's funny how one already needs to dig a
little even to piece together very recent very electronically recorded
history, though of course one can always consult the actual people. :)

Regards,

Martijn

From faassen at startifact.com  Tue Feb  7 17:14:28 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Tue, 07 Feb 2012 17:14:28 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <4F30FB83.3080706@egenix.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<4F30B686.6040101@zopyx.com>
	<20120207101609.GA12442@sleipnir.bytereef.org>
	<4F30FB83.3080706@egenix.com>
Message-ID: <jgril4$5q1$1@dough.gmane.org>

On 02/07/2012 11:22 AM, M.-A. Lemburg wrote:
> Guys, can we please stop this flame war and agree to disagree ?!
>
> The current discussion is not leading anywhere.

+1

Regards,

Martijn




From faassen at startifact.com  Tue Feb  7 17:18:58 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Tue, 07 Feb 2012 17:18:58 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgqfnd$vvj$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgqfnd$vvj$1@dough.gmane.org>
Message-ID: <jgriti$95t$1@dough.gmane.org>

On 02/07/2012 07:18 AM, Kai Diefenbach wrote:
> On 2012-02-06 21:58:11 +0000, Terry Reedy said:
>
>> On 2/6/2012 3:17 PM, Andreas Jung wrote:
>>> My point about this: if a person does not want
>>> to host its package on PyPi than it should stay away from PyPI.
>>
>> The Python Package Index was originally just that: a package *INDEX*,
>> aiming to be a complete index.
>
> If a listed package is not available (because an external server is
> down) the index is broken.

That's an interesting observation. I would think 'broken' is strong 
language, but it the index can at least be considered incorrect in that 
particular instance.

If people have tools that rely on the index being correct, then this it 
being incorrect can be a problem. You can either say those tools 
shouldn't be used for "real" development work ("you're doing it wrong"), 
or encourage people to provide the package on PyPI as well 
(encouragement as a social solution), or consider facilities to provide 
redundancy (caching, mirroring) to help with the experience (a technical 
solution).

Regards,

Martijn


From pje at telecommunity.com  Tue Feb  7 18:02:11 2012
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 7 Feb 2012 12:02:11 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <4F303561.6010404@zopyx.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
Message-ID: <CALeMXf6vqkaE++70VUndV-At6pti4FGURyPbKAv0VRRjmToc7w@mail.gmail.com>

On Mon, Feb 6, 2012 at 3:17 PM, Andreas Jung <lists at zopyx.com> wrote:

> My point about this: if a person does not want
> to host its package on PyPi than it should stay away from PyPI. Package
> hygiene and a certain level of professional package repository is more
> important and personal reasons for not hosting packages on PyPI.
>

Note that PyPI is also used to publish metadata about packages which are in
development and only available in snapshot releases or revision control
systems.  So the "it shouldn't be hosted elsewhere" argument doesn't really
wash.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120207/2e7d540e/attachment.html>

From pje at telecommunity.com  Tue Feb  7 18:05:46 2012
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 7 Feb 2012 12:05:46 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgriti$95t$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgqfnd$vvj$1@dough.gmane.org> <jgriti$95t$1@dough.gmane.org>
Message-ID: <CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>

On Tue, Feb 7, 2012 at 11:18 AM, Martijn Faassen <faassen at startifact.com>wrote:

> On 02/07/2012 07:18 AM, Kai Diefenbach wrote:
>
>> If a listed package is not available (because an external server is
>> down) the index is broken.
>>
>
> That's an interesting observation. I would think 'broken' is strong
> language, but it the index can at least be considered incorrect in that
> particular instance.
>
> If people have tools that rely on the index being correct, then this it
> being incorrect can be a problem. You can either say those tools shouldn't
> be used for "real" development work ("you're doing it wrong"), or encourage
> people to provide the package on PyPI as well (encouragement as a social
> solution), or consider facilities to provide redundancy (caching,
> mirroring) to help with the experience (a technical solution).
>

Note, too, that prior to setuptools' development, there wasn't even any
expectation that projects listed on PyPI even have a current *release*, or
even have any *source code written* , let alone packages available for
download from PyPI itself.  (PyPI uploading was developed around the same
time as the first versions of setuptools and EasyInstall.)

Just because the common use-case for PyPI nowadays is to pull down
installation files, doesn't mean the previous use cases which PyPI catered
to are gone or not worth supporting any more.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120207/69e2bbc7/attachment.html>

From donald.stufft at gmail.com  Tue Feb  7 18:06:15 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 7 Feb 2012 12:06:15 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CALeMXf6vqkaE++70VUndV-At6pti4FGURyPbKAv0VRRjmToc7w@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
	<CALeMXf6vqkaE++70VUndV-At6pti4FGURyPbKAv0VRRjmToc7w@mail.gmail.com>
Message-ID: <17E36799ADFA466F815FE9788BE8430F@gmail.com>

On Tuesday, February 7, 2012 at 12:02 PM, PJ Eby wrote:
> On Mon, Feb 6, 2012 at 3:17 PM, Andreas Jung <lists at zopyx.com (mailto:lists at zopyx.com)> wrote:
> > My point about this: if a person does not want to host its package on PyPi than it should stay away from PyPI. Package
> > hygiene and a certain level of professional package repository is more
> > important and personal reasons for not hosting packages on PyPI.
> 
> Note that PyPI is also used to publish metadata about packages which are in development and only available in snapshot releases or revision control systems.  So the "it shouldn't be hosted elsewhere" argument doesn't really wash.'
This is a matter of opinion really, Personally I think if your package is in development you should publish snapshot releases to PyPI. But even then this is really a special case for packages that don't have real releases yet. For packages with real releases you can do ==dev for those. 
> 
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120207/b7238014/attachment-0001.html>

From lists at zopyx.com  Tue Feb  7 18:39:32 2012
From: lists at zopyx.com (Andreas Jung)
Date: Tue, 07 Feb 2012 18:39:32 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CALeMXf6vqkaE++70VUndV-At6pti4FGURyPbKAv0VRRjmToc7w@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
	<CALeMXf6vqkaE++70VUndV-At6pti4FGURyPbKAv0VRRjmToc7w@mail.gmail.com>
Message-ID: <4F3161D4.5000106@zopyx.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



PJ Eby wrote:
> On Mon, Feb 6, 2012 at 3:17 PM, Andreas Jung <lists at zopyx.com 
> <mailto:lists at zopyx.com>> wrote:
> 
> My point about this: if a person does not want to host its package on
> PyPi than it should stay away from PyPI. Package hygiene and a
> certain level of professional package repository is more important
> and personal reasons for not hosting packages on PyPI.
> 
> 
> Note that PyPI is also used to publish metadata about packages which
> are in development and only available in snapshot releases or
> revision control systems.  So the "it shouldn't be hosted elsewhere"
> argument doesn't really wash.
> 

Development packages and snapshot don't belong on PyPI in my opinion.
PyPI should not be an unflushed package toilet. The Perl world has
a working repository infrastructure with CPAN since many, many years.
Python is years behind.

- -aj
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJPMWHUAAoJEADcfz7u4AZj0AELv0h/WhGbOP2wHB+PF7XS4/WY
/7F99mvXmzfYs31MTJQizR8CK+B9d6n5ebsthCoeBE1pEoRK/F3wnvTsC2PwiyRR
QSwSXACXJRrqk+BT/GTSV5Sn8iS2HkTdaOd3BKyHopadbdJMuRYA6+jh8clTUZ8H
v3d4+GLN23YI0QDBZ6nke7MlPXTCfIN0htlhi3IWQwpMwT6ptZwDP/eNXIGAJGj1
jtuDWVgwGBwAVq8h8pMew6Us8ydXk4zmzG6BFQ9CU2jDbUdhVVK6heouEETRm7ae
NNWEfxAlF6c2mrhA//I9hdR5dGPnvr+GQcSp/tU5xXOkdjfu3S3oKgipEUQIaC/C
i9aIaqsFpReZa9J5NBZFYXGYILHEOCGjwD5TJIeIl2DAOdYBZHrZuO9sR1/4NQIJ
xrNr8+BygQVj/gxt7Up8RUgDP95KvgOK6VUxatSLXd1yNZfKtzc4tlPX4Je45Rpx
8BNCAWzKltUZItr573U2IKffZORoovc=
=drFc
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120207/5eb6063b/attachment.vcf>

From faassen at startifact.com  Tue Feb  7 18:44:50 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Tue, 7 Feb 2012 18:44:50 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgqfnd$vvj$1@dough.gmane.org> <jgriti$95t$1@dough.gmane.org>
	<CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>
Message-ID: <CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>

Hi there,

On Tue, Feb 7, 2012 at 6:05 PM, PJ Eby <pje at telecommunity.com> wrote:
> Note, too, that prior to setuptools' development, there wasn't even any
> expectation that projects listed on PyPI even have a current *release*,
> or even have any *source code written* ,

That a package that only exists in a version control system can be
listed on PyPI makes sense to me. That a package without any source
code is listed on PyPI makes less sense to me; are there examples
where that does or did make sense?

But if at some point there *was* a release of the package listed in
PyPI and now, without author intent involved (to leave out the moral
arguments) the release cannot be found anymore, I'd say PyPI is
incorrect.

There are two possible cases:

* release cannot be found at all anymore by humans clicking around
PyPI; it's not in PyPI nor in any linked sites.

* release can be found by a determined human in linked sites, but
automated tools now fail to download the case where they did not
previously.

The first case you could argue is a flaw in the tools and not an
incorrect index, though you could also argue that tools shouldn't be
based on unreliable magic behavior, so you could argue the index isn't
providing one of the services that people rely on.

The second case is an actual incorrectness in the index.

> Just because the common use-case for PyPI nowadays is to pull down
> installation files, doesn't mean the previous use cases which PyPI catered
> to are gone or not worth supporting any more.

I wasn't arguing otherwise myself.

But I am interested in approaches that make releases that were listed
on PyPI and were accessible by automated tools permanently accessible
by such tools (unless author intent is involved; I want to leave that
off the table in this discussion).

Regards,

Martijn

From fred at fdrake.net  Tue Feb  7 19:18:24 2012
From: fred at fdrake.net (Fred Drake)
Date: Tue, 7 Feb 2012 13:18:24 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org>
	<jgmgrf$ndi$1@dough.gmane.org> <jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
	<jgpie0$a8s$1@dough.gmane.org> <jgqfnd$vvj$1@dough.gmane.org>
	<jgriti$95t$1@dough.gmane.org>
	<CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>
	<CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>
Message-ID: <CAFT4OTE6DkBVPZ=Zr8GTm=aFBqOELYY3cA16gPSFOctCxNYpfQ@mail.gmail.com>

On Tue, Feb 7, 2012 at 12:44 PM, Martijn Faassen <faassen at startifact.com> wrote:
> That a package without any source code is listed on PyPI makes
> less sense to me; are there examples where that does or did make sense?

Originally, one purpose of the index was to reserve names in order to
avoid future conflicts.  This was a larger issue before the now-common use
of namespace packages evolved.


  -Fred

-- 
Fred L. Drake, Jr.? ? <fred at fdrake.net>
"A storm broke loose in my mind."? --Albert Einstein

From pje at telecommunity.com  Tue Feb  7 19:58:04 2012
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 7 Feb 2012 13:58:04 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <17E36799ADFA466F815FE9788BE8430F@gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
	<CALeMXf6vqkaE++70VUndV-At6pti4FGURyPbKAv0VRRjmToc7w@mail.gmail.com>
	<17E36799ADFA466F815FE9788BE8430F@gmail.com>
Message-ID: <CALeMXf4K76Rhu_RD710dhTv=4D+N-cN9ThSdJwLavn1TLjYQYA@mail.gmail.com>

On Tue, Feb 7, 2012 at 12:06 PM, Donald Stufft <donald.stufft at gmail.com>wrote:

> On Tuesday, February 7, 2012 at 12:02 PM, PJ Eby wrote:
>
> On Mon, Feb 6, 2012 at 3:17 PM, Andreas Jung <lists at zopyx.com> wrote:
>
> My point about this: if a person does not want
> to host its package on PyPi than it should stay away from PyPI. Package
> hygiene and a certain level of professional package repository is more
> important and personal reasons for not hosting packages on PyPI.
>
>
> Note that PyPI is also used to publish metadata about packages which are
> in development and only available in snapshot releases or revision control
> systems.  So the "it shouldn't be hosted elsewhere" argument doesn't really
> wash.'
>
> This is a matter of opinion really, Personally I think if your package is
> in development you should publish snapshot releases to PyPI.
>

Yes, but now we get into the wonderful world of how many releases do you
actually want active vs. hidden vs. deleted, and now there are that many
more files to be possible frozen and mirrored and archived and whatnot,
which isn't really suitable for such dev releases.

(Also, in the specific case of my snapshot-only packages, I have automated
builds that keep a rotating set of snapshots in a server-local download
directory for public access; I wouldn't want that build process
automatically uploading that stuff to PyPI, as it adds more moving parts
for things to break on my end.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120207/dd9d3564/attachment.html>

From pje at telecommunity.com  Tue Feb  7 20:01:00 2012
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 7 Feb 2012 14:01:00 -0500
Subject: [Catalog-sig] Distutils sdist formats best practice
In-Reply-To: <jgp22g$2lq$1@dough.gmane.org>
References: <jgp22g$2lq$1@dough.gmane.org>
Message-ID: <CALeMXf5GjqtK1isjrdqULk20MQVAZXPeviBHFLLDziPN10B76g@mail.gmail.com>

On Mon, Feb 6, 2012 at 12:19 PM, Alex Clark <aclark at aclark.net> wrote:

> What do pip/easy_install/etc do when they encounter both a .zip and a
> .tar.gz, for example?


IIRC, easy_install will take the longer filename in preference to the
shorter one, all else being equal; that's its final tiebreaker after what
kind of thing it expects to find at a given URL.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120207/32d47b81/attachment.html>

From martin at v.loewis.de  Tue Feb  7 20:39:34 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Feb 2012 20:39:34 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>	<jgm0lt$8bl$1@dough.gmane.org>
	<jgmgrf$ndi$1@dough.gmane.org>	<jgorrb$c8m$1@dough.gmane.org>	<20120206200823.GA8910@sleipnir.bytereef.org>	<4F303561.6010404@zopyx.com>
	<jgpie0$a8s$1@dough.gmane.org>	<jgqfnd$vvj$1@dough.gmane.org>
	<jgriti$95t$1@dough.gmane.org>	<CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>
	<CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>
Message-ID: <4F317DF6.3070704@v.loewis.de>

> That a package that only exists in a version control system can be
> listed on PyPI makes sense to me. That a package without any source
> code is listed on PyPI makes less sense to me; are there examples
> where that does or did make sense?

I think there are still a number of packages listed on PyPI that
are not free software, so the only way to "download" them is to pay
for a license (and you may then get a CD-ROM instead of a download).

I consider that a reasonable use case (despite encouraging software
developers to develop all their software as free software).

> But if at some point there *was* a release of the package listed in
> PyPI and now, without author intent involved (to leave out the moral
> arguments) the release cannot be found anymore, I'd say PyPI is
> incorrect.

Very true. However, PyPI is always incorrect in many respects: the
package descriptions may be incorrect or incomplete, the classifiers
may be incorrect, and author information may become out of date after
some time.

By design, getting correct information into the index is the task
of the package authors, not of the PyPI infrastructure. The only
exception are cases where the information is maliciously misleading,
and then the information is deleted, not corrected.

Regards,
Martin

From faassen at startifact.com  Tue Feb  7 20:51:19 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Tue, 7 Feb 2012 20:51:19 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <4F317DF6.3070704@v.loewis.de>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgqfnd$vvj$1@dough.gmane.org> <jgriti$95t$1@dough.gmane.org>
	<CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>
	<CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>
	<4F317DF6.3070704@v.loewis.de>
Message-ID: <CAGT4ZFi=T-q5=wkmwL3xcNPX2m4188X16g+fudUO7SbKMLJEmg@mail.gmail.com>

Hi there,

On Tue, Feb 7, 2012 at 8:39 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> That a package that only exists in a version control system can be
>> listed on PyPI makes sense to me. That a package without any source
>> code is listed on PyPI makes less sense to me; are there examples
>> where that does or did make sense?
>
> I think there are still a number of packages listed on PyPI that
> are not free software, so the only way to "download" them is to pay
> for a license (and you may then get a CD-ROM instead of a download).
>
> I consider that a reasonable use case (despite encouraging software
> developers to develop all their software as free software).

Sure, but PJE said "without any source code *written*", a commercially
available package would still have source code somewhere presumably.
:)

>> But if at some point there *was* a release of the package listed in
>> PyPI and now, without author intent involved (to leave out the moral
>> arguments) the release cannot be found anymore, I'd say PyPI is
>> incorrect.
>
> Very true. However, PyPI is always incorrect in many respects: the
> package descriptions may be incorrect or incomplete, the classifiers
> may be incorrect, and author information may become out of date after
> some time.

Yes, that's true. I guess there are different levels of incorrectness.

> By design, getting correct information into the index is the task
> of the package authors, not of the PyPI infrastructure. The only
> exception are cases where the information is maliciously misleading,
> and then the information is deleted, not corrected.

I'm talking about the scenario where the authors did get correct
information into the index, and
information such as package description, authors, classifiers etc are
still all correct, and the only thing that broke is
the download link. I.e. all the information *managed* by PyPI is
correct but what PyPI is pointing to isn't any more. This is an
incorrectness that isn't under PyPI's control. It's similar to the
situation where a website URL that a PyPI page points to breaking, but
with more serious consequences for tools. This is because release
files are a bit more like package metadata than they are like links.

Regards,

Martijn

From martin at v.loewis.de  Tue Feb  7 22:31:50 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Feb 2012 22:31:50 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CAGT4ZFi=T-q5=wkmwL3xcNPX2m4188X16g+fudUO7SbKMLJEmg@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>	<jgm0lt$8bl$1@dough.gmane.org>	<jgmgrf$ndi$1@dough.gmane.org>	<jgorrb$c8m$1@dough.gmane.org>	<20120206200823.GA8910@sleipnir.bytereef.org>	<4F303561.6010404@zopyx.com>	<jgpie0$a8s$1@dough.gmane.org>	<jgqfnd$vvj$1@dough.gmane.org>	<jgriti$95t$1@dough.gmane.org>	<CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>	<CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>	<4F317DF6.3070704@v.loewis.de>
	<CAGT4ZFi=T-q5=wkmwL3xcNPX2m4188X16g+fudUO7SbKMLJEmg@mail.gmail.com>
Message-ID: <4F319846.8090806@v.loewis.de>

> I'm talking about the scenario where the authors did get correct
> information into the index, and
> information such as package description, authors, classifiers etc are
> still all correct, and the only thing that broke is
> the download link. I.e. all the information *managed* by PyPI is
> correct but what PyPI is pointing to isn't any more. This is an
> incorrectness that isn't under PyPI's control. It's similar to the
> situation where a website URL that a PyPI page points to breaking, but
> with more serious consequences for tools. This is because release
> files are a bit more like package metadata than they are like links.

I agree that it's more serious to the users (although I find a stale
home page or an incorrect contact email address fairly serious too).

I still maintain that none of this incorrectness is PyPI's "fault",
or that we should feel responsible for fixing it. It's just a storage
of information, with no responsibility on python.org's side except to
preserve the data (within legal and moral boundaries).

If you, for some reason, don't like packages who don't upload their
source code to PyPI, then don't use them. Out of principle, I personally
don't use non-free software if there is a free alternative available
(regardless where it's hosted), but I wouldn't demand that all software
on PyPI must be free.

Likewise, you really should accept the status quo with respect to file
hosting as a good thing: people apparently *want* to do things in
different ways, also in ways that you consider unprofessional and
hurtful to the users of the software. Just don't use such packages
if you don't like them.

Regards,
Martin

From ubershmekel at gmail.com  Tue Feb  7 23:29:15 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 8 Feb 2012 00:29:15 +0200
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <4F319846.8090806@v.loewis.de>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgqfnd$vvj$1@dough.gmane.org> <jgriti$95t$1@dough.gmane.org>
	<CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>
	<CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>
	<4F317DF6.3070704@v.loewis.de>
	<CAGT4ZFi=T-q5=wkmwL3xcNPX2m4188X16g+fudUO7SbKMLJEmg@mail.gmail.com>
	<4F319846.8090806@v.loewis.de>
Message-ID: <CANSw7Kxp5=8CPB45pWBrZGN66n-PQOi8UqQn4fkYeE=hjygE_Q@mail.gmail.com>

On Tue, Feb 7, 2012 at 11:31 PM, "Martin v. L?wis" <martin at v.loewis.de>wrote:

> > I'm talking about the scenario where the authors did get correct
> > information into the index, and
> > information such as package description, authors, classifiers etc are
> > still all correct, and the only thing that broke is
> > the download link. I.e. all the information *managed* by PyPI is
> > correct but what PyPI is pointing to isn't any more. This is an
> > incorrectness that isn't under PyPI's control. It's similar to the
> > situation where a website URL that a PyPI page points to breaking, but
> > with more serious consequences for tools. This is because release
> > files are a bit more like package metadata than they are like links.
>
> I agree that it's more serious to the users (although I find a stale
> home page or an incorrect contact email address fairly serious too).
>
> I still maintain that none of this incorrectness is PyPI's "fault",
> or that we should feel responsible for fixing it. It's just a storage
> of information, with no responsibility on python.org's side except to
> preserve the data (within legal and moral boundaries).
>
>
I think we all agree we should try and remove dead links or at least mark
them as such. Google/wikipedia/download.com/etc wouldn't serve dead links
in search results and I don't think we should treat PyPI as the Python
Packaging Archive.

I don't know how that can be done for contact email addresses but any url
pypi points to (package host or website) should be checked once in a while
imho.

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120208/c2a8251c/attachment.html>

From martin at v.loewis.de  Wed Feb  8 01:21:52 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 08 Feb 2012 01:21:52 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CANSw7Kxp5=8CPB45pWBrZGN66n-PQOi8UqQn4fkYeE=hjygE_Q@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgqfnd$vvj$1@dough.gmane.org> <jgriti$95t$1@dough.gmane.org>
	<CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>
	<CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>
	<4F317DF6.3070704@v.loewis.de>
	<CAGT4ZFi=T-q5=wkmwL3xcNPX2m4188X16g+fudUO7SbKMLJEmg@mail.gmail.com>
	<4F319846.8090806@v.loewis.de>
	<CANSw7Kxp5=8CPB45pWBrZGN66n-PQOi8UqQn4fkYeE=hjygE_Q@mail.gmail.com>
Message-ID: <20120208012152.Horde.b3GaSUlCcOxPMcAgA_OBj0A@webmail.df.eu>

> I think we all agree we should try and remove dead links or at least mark
> them as such.

I don't know who "we" is that should try to remove dead links,
and whose dead links. I certainly don't agree that I (as a PyPI operator)
should remove any links on any package (except for legal or moral reasons).

> Google/wikipedia/download.com/etc wouldn't serve dead links
> in search results and I don't think we should treat PyPI as the Python
> Packaging Archive.

Why not? People certainly use it as an archive - and the same people  
suggesting
that PyPI should host all files also insist that old releases should never be
deleted from PyPI (making it an archive).

> I don't know how that can be done for contact email addresses but any url
> pypi points to (package host or website) should be checked once in a while
> imho.

Feel free to do so, and to then contact the package authors to update their
release pages.

Regards,
Martin



From tjreedy at udel.edu  Wed Feb  8 02:53:28 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 07 Feb 2012 20:53:28 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <jgqfnd$vvj$1@dough.gmane.org>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgqfnd$vvj$1@dough.gmane.org>
Message-ID: <jgskj7$8el$1@dough.gmane.org>

On 2/7/2012 1:18 AM, Kai Diefenbach wrote:
> On 2012-02-06 21:58:11 +0000, Terry Reedy said:
>
>> On 2/6/2012 3:17 PM, Andreas Jung wrote:
>>> My point about this: if a person does not want
>>> to host its package on PyPi than it should stay away from PyPI.
>>
>> The Python Package Index was originally just that: a package *INDEX*,
>> aiming to be a complete index.
>
> If a listed package is not available (because an external server is
> down) the index is broken.

If a package has files on pypi, there is an html table with these headers

File 	Type 	Py Version 	Uploaded on 	Size 	# downloads

If it does not, there is no such table. Any 'professional' should be 
able to notice the difference and ignore packages without. Any external 
software should be able to detect same.

If you want, request a new entry under 'Browse packages' for 'Packages 
in repository'.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Wed Feb  8 03:23:46 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 07 Feb 2012 21:23:46 -0500
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <CALeMXf6vqkaE++70VUndV-At6pti4FGURyPbKAv0VRRjmToc7w@mail.gmail.com>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com>
	<CALeMXf6vqkaE++70VUndV-At6pti4FGURyPbKAv0VRRjmToc7w@mail.gmail.com>
Message-ID: <jgsmc0$j0o$1@dough.gmane.org>

On 2/7/2012 12:02 PM, PJ Eby wrote:

> Note that PyPI is also used to publish metadata about packages which are
> in development and only available in snapshot releases or revision
> control systems.

Thank you for reminding me. That is what I will do once I put my project 
in a publicly accessible RCS.

-- 
Terry Jan Reedy


From faassen at startifact.com  Wed Feb  8 14:21:20 2012
From: faassen at startifact.com (Martijn Faassen)
Date: Wed, 8 Feb 2012 14:21:20 +0100
Subject: [Catalog-sig] What is the point of pythonpackages.com?
In-Reply-To: <20120208012152.Horde.b3GaSUlCcOxPMcAgA_OBj0A@webmail.df.eu>
References: <20120204194054.GA29629@sleipnir.bytereef.org>
	<jgm0lt$8bl$1@dough.gmane.org> <jgmgrf$ndi$1@dough.gmane.org>
	<jgorrb$c8m$1@dough.gmane.org>
	<20120206200823.GA8910@sleipnir.bytereef.org>
	<4F303561.6010404@zopyx.com> <jgpie0$a8s$1@dough.gmane.org>
	<jgqfnd$vvj$1@dough.gmane.org> <jgriti$95t$1@dough.gmane.org>
	<CALeMXf6+WFZRuMAJK-SdRw5272_BP5Rkce-93WjH7=fnF99jZQ@mail.gmail.com>
	<CAGT4ZFhZb0LzW9v9PwKnreXNETP1Pj_+DzZsWUgPAxWmd7f05w@mail.gmail.com>
	<4F317DF6.3070704@v.loewis.de>
	<CAGT4ZFi=T-q5=wkmwL3xcNPX2m4188X16g+fudUO7SbKMLJEmg@mail.gmail.com>
	<4F319846.8090806@v.loewis.de>
	<CANSw7Kxp5=8CPB45pWBrZGN66n-PQOi8UqQn4fkYeE=hjygE_Q@mail.gmail.com>
	<20120208012152.Horde.b3GaSUlCcOxPMcAgA_OBj0A@webmail.df.eu>
Message-ID: <CAGT4ZFhXJP5MSNXCjdNoDLN-KgCc6bAQ00oYe4d5o0UJY0nQLA@mail.gmail.com>

Hi there,

On Wed, Feb 8, 2012 at 1:21 AM,  <martin at v.loewis.de> wrote:
[snip]
> Why not? People certainly use it as an archive - and the same people
> suggesting
> that PyPI should host all files also insist that old releases should never
> be deleted from PyPI (making it an archive).

There are various semantics of the word "archive" involved.

One way to see an archive is as a repository of out-of-date historical
information. Its only interest is historical. Broken links are
acceptable in that.

Another way to see an archive is as a repository of information that
is current. While some bits (releases) were added to the archive long
ago, they still see active use every day.

(There's a grey area. Python 1.5.2 is historical to most of us, but is
undoubtedly still in active use somewhere. If it were to disappear
from python.org the people complaining about it would have a
legitimate complaint: it's unnecessarily hurting its users to do so, I
think, for little gain to the python.org maintainers. But for most of
us the 1.5.2 download page is of historical value only.)

PyPI is both: it's both an archive of historical information and
that's why links in its metadata and documentation should be allowed
to remain even when the outside world has changed and they are broken,
and it's a repository of current information, where we want the
metadata and in particular the *releases* to remain available.

If an archive contained no links to the outside world at all, an
archive (active action to modify the archive disregarded) would
automatically be both historical and current. But PyPI does contain
links, and in particular links to releases.

The thing that brings tension between the two uses of PyPI is that
releases are, in the "repository of current information" sense, more
like metadata than like links. PyPI retains old metadata, but *links*
to releases can break. So if a release is uploaded to PyPI, the
release will remain (unless active action is taken), and this
permanence is under control of PyPI, just like that of metadata.  If a
*link* is uploaded to PyPI for indicating releases, it can only be
maintained by PyPI in the "historical archive" sense; it might be up
to date or might be outdated, and PyPI cannot help it or control it.

If PyPI *only* contained links to releases and didn't contain releases
itself, we would either not have the automatic download tools we have
now, or we'd have cache or repository technology to make sure that
releases *can* be reliably accessed to reduce the points of failure.
If PyPI *only* contained releases and no links to releases, we'd not
be having this discussion (we'd only have the discussion about people
actively removing old releases). But PyPI does both and that's what is
creating complexity.

I think the best way out would be for caching technology for active
releases to find active use (if the license information in the PyPI
metadata allows such caching). This is a technical solution that can
be worked on independently.

Regards,

Martijn

From techtonik at gmail.com  Sat Feb 18 22:01:47 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Sun, 19 Feb 2012 00:01:47 +0300
Subject: [Catalog-sig] Fwd: [ pypi-Bugs-3488989 ] Need a way to list all
	package versions
In-Reply-To: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
Message-ID: <CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>

Hi,

I find SF tracker very annoying, because it doesn't even allow to add
comments to a closed issue (see below). That's why crossposting here.

So, there two questions:
1. does anybody else also thinks that having a way to list all package
versions available on PyPI would be awesome?
2. is there people who don't like SF tracker?

Please, CC.


---------- Forwarded message ----------
From: SourceForge.net <noreply at sourceforge.net>
Date: Sat, Feb 18, 2012 at 8:14 PM
Subject: [ pypi-Bugs-3488989 ] Need a way to list all package versions
To: "SourceForge.net" <noreply at sourceforge.net>


Bugs item #3488989, was opened at 2012-02-18 03:58
Message generated for change (Comment added) made by loewis
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=513503&aid=3488989&group_id=66150

Please note that this message will contain a full copy of the comment
thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Closed
>Resolution: Works For Me
Priority: 5
Private: No
Submitted By: anatoly techtonik (techtonik)
Assigned to: Nobody/Anonymous (nobody)
Summary: Need a way to list all package versions

Initial Comment:
Web interface doesn't have a feature to list all package version available
from PyPI. This can be handy.

----------------------------------------------------------------------

>Comment By: Martin v. L?wis (loewis)
Date: 2012-02-18 09:14

Message:
You cannot. This is intentional; the Sphinx authors chose to hide all but
the current version from you.

Closing the report as "works for me"; PyPI does support listing multiple
versions, and the feature seems to work correctly as designed.

----------------------------------------------------------------------

Comment By: anatoly techtonik (techtonik)
Date: 2012-02-18 06:47

Message:
All right. So, how can I list all Sphinx versions available from PyPI using
web interface of Python Package Index?

----------------------------------------------------------------------

Comment By: Martin v. L?wis (loewis)
Date: 2012-02-18 06:20

Message:
Becausse only a single releae of Sphinx is visible.

----------------------------------------------------------------------

Comment By: anatoly techtonik (techtonik)
Date: 2012-02-18 05:26

Message:
Why I can't see anything for Sphinx?

http://pypi.python.org/pypi/Sphinx

----------------------------------------------------------------------

Comment By: Martin v. L?wis (loewis)
Date: 2012-02-18 04:11

Message:
It most certainly supports listing all versions. E.g. if you go to

http://pypi.python.org/pypi/Django

you see all Django versions.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=513503&aid=3488989&group_id=66150
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120219/74fb7cc9/attachment.html>

From donald.stufft at gmail.com  Sat Feb 18 22:07:28 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Sat, 18 Feb 2012 16:07:28 -0500
Subject: [Catalog-sig] Fwd: [ pypi-Bugs-3488989 ] Need a way to list all
 package versions
In-Reply-To: <CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
Message-ID: <0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>

On Saturday, February 18, 2012 at 4:01 PM, anatoly techtonik wrote:
> Hi,
>  
> I find SF tracker very annoying, because it doesn't even allow to add comments to a closed issue (see below). That's why crossposting here.
>  
> So, there two questions:  
> 1. does anybody else also thinks that having a way to list all package versions available on PyPI would be awesome?
>  
>  

It's possible via the API to list all packages regardless of hidden or not. The Web UI doesn't support it. I'm assuming from the ticket PyPI is happy with the way it works, but fwiw you can see all the package versions for Sphinx at http://crate.io/packages/Sphinx/  (click the All Versions tab).  
> 2. is there people who don't like SF tracker?
>  
>  

Haven't really used the SF tracker, but if it doesn't allow comments on closed issues that would be mildly annoying to me.  
>   
>  
>  

>  
> Please, CC.
>  
>  
> ---------- Forwarded message ----------
> From: SourceForge.net (http://SourceForge.net) <noreply at sourceforge.net (mailto:noreply at sourceforge.net)>
> Date: Sat, Feb 18, 2012 at 8:14 PM
> Subject: [ pypi-Bugs-3488989 ] Need a way to list all package versions
> To: "SourceForge.net (http://SourceForge.net)" <noreply at sourceforge.net (mailto:noreply at sourceforge.net)>
>  
>  
> Bugs item #3488989, was opened at 2012-02-18 03:58
> Message generated for change (Comment added) made by loewis
> You can respond by visiting:
> https://sourceforge.net/tracker/?func=detail&atid=513503&aid=3488989&group_id=66150
>  
> Please note that this message will contain a full copy of the comment thread,
> including the initial issue submission, for this request,
> not just the latest update.
> Category: None
> Group: None
> >Status: Closed
> >Resolution: Works For Me
> Priority: 5
> Private: No
> Submitted By: anatoly techtonik (techtonik)
> Assigned to: Nobody/Anonymous (nobody)
> Summary: Need a way to list all package versions
>  
> Initial Comment:
> Web interface doesn't have a feature to list all package version available from PyPI. This can be handy.
>  
> ----------------------------------------------------------------------
>  
> >Comment By: Martin v. L?wis (loewis)
> Date: 2012-02-18 09:14
>  
> Message:
> You cannot. This is intentional; the Sphinx authors chose to hide all but
> the current version from you.
>  
> Closing the report as "works for me"; PyPI does support listing multiple
> versions, and the feature seems to work correctly as designed.
>  
> ----------------------------------------------------------------------
>  
> Comment By: anatoly techtonik (techtonik)
> Date: 2012-02-18 06:47
>  
> Message:
> All right. So, how can I list all Sphinx versions available from PyPI using
> web interface of Python Package Index?
>  
> ----------------------------------------------------------------------
>  
> Comment By: Martin v. L?wis (loewis)
> Date: 2012-02-18 06:20
>  
> Message:
> Becausse only a single releae of Sphinx is visible.
>  
> ----------------------------------------------------------------------
>  
> Comment By: anatoly techtonik (techtonik)
> Date: 2012-02-18 05:26
>  
> Message:
> Why I can't see anything for Sphinx?
>  
> http://pypi.python.org/pypi/Sphinx
>  
> ----------------------------------------------------------------------
>  
> Comment By: Martin v. L?wis (loewis)
> Date: 2012-02-18 04:11
>  
> Message:
> It most certainly supports listing all versions. E.g. if you go to
>  
> http://pypi.python.org/pypi/Django
>  
> you see all Django versions.
>  
> ----------------------------------------------------------------------
>  
> You can respond by visiting:
> https://sourceforge.net/tracker/?func=detail&atid=513503&aid=3488989&group_id=66150
>  
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
>  
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120218/77a1b95a/attachment.html>

From martin at v.loewis.de  Sat Feb 18 22:26:40 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Sat, 18 Feb 2012 22:26:40 +0100
Subject: [Catalog-sig] Fwd: [ pypi-Bugs-3488989 ] Need a way to list all
 package versions
In-Reply-To: <0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
	<0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>
Message-ID: <20120218222640.Horde.tZ9TIML8999PQBeQ1QHlgtA@webmail.df.eu>

> Haven't really used the SF tracker, but if it doesn't allow comments  
> on closed issues that would be mildly annoying to me.

It's an per-issue option available to developers; I chose to close this
issue for comments. I could have allowed follow-ups, so this wasn't SF's
choice.

Regards,
Martin


From hanno at hannosch.eu  Sun Feb 19 00:57:35 2012
From: hanno at hannosch.eu (Hanno Schlichting)
Date: Sun, 19 Feb 2012 00:57:35 +0100
Subject: [Catalog-sig] Fwd: [ pypi-Bugs-3488989 ] Need a way to list all
 package versions
In-Reply-To: <0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
	<0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>
Message-ID: <CAJ5sox4mOp6hZJ9vE+t5pG9Ht3GZUYVeOkSi66-Yy4+624ygow@mail.gmail.com>

On Sat, Feb 18, 2012 at 10:07 PM, Donald Stufft <donald.stufft at gmail.com> wrote:
> It's possible via the API to list all packages regardless of hidden or not.
> The Web UI doesn't support it. I'm assuming from the ticket PyPI is happy
> with the way it works, but fwiw you can see all the package versions for
> Sphinx at?http://crate.io/packages/Sphinx/? (click the All Versions tab).

You can always look at the index view at for example
http://pypi.python.org/simple/Sphinx/ - though this view is made for
tools and not humans ;)

If you are only interested in downloads hosted on PyPi see for example
http://pypi.python.org/packages/source/S/Sphinx/

Hanno

From pydanny at gmail.com  Sun Feb 19 01:31:22 2012
From: pydanny at gmail.com (Daniel Greenfeld)
Date: Sat, 18 Feb 2012 16:31:22 -0800
Subject: [Catalog-sig] Fwd: [ pypi-Bugs-3488989 ] Need a way to list all
 package versions
In-Reply-To: <CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
Message-ID: <CAOoSJ_rEUc6XxGDsgjmbK2f1Vj-zZTwe0Kw8GNTbSUZLcP-bjw@mail.gmail.com>

The documentation and API to solve #1 has been available for a long
time. We've used the PyPI API on hidden packages to do counts on for
Django Packages for 1.5 years. It's a pretty easy query.

Danny

On Sat, Feb 18, 2012 at 1:01 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> Hi,
>
> I find SF tracker very annoying, because it doesn't even allow to add
> comments to a closed issue (see below). That's why crossposting here.
>
> So, there two questions:
> 1. does anybody else also thinks that?having a way to list?all package
> versions available on PyPI would be awesome?
> 2. is there people who don't like SF tracker?
>
> Please, CC.
>
>
> ---------- Forwarded message ----------
> From: SourceForge.net <noreply at sourceforge.net>
> Date: Sat, Feb 18, 2012 at 8:14 PM
> Subject: [ pypi-Bugs-3488989 ] Need a way to list all package versions
> To: "SourceForge.net" <noreply at sourceforge.net>
>
>
> Bugs item #3488989, was opened at 2012-02-18 03:58
> Message generated for change (Comment added) made by loewis
> You can respond by visiting:
> https://sourceforge.net/tracker/?func=detail&atid=513503&aid=3488989&group_id=66150
>
> Please note that this message will contain a full copy of the comment
> thread,
> including the initial issue submission, for this request,
> not just the latest update.
> Category: None
> Group: None
>>Status: Closed
>>Resolution: Works For Me
> Priority: 5
> Private: No
> Submitted By: anatoly techtonik (techtonik)
> Assigned to: Nobody/Anonymous (nobody)
> Summary: Need a way to list all package versions
>
> Initial Comment:
> Web interface doesn't have a feature to list all package version available
> from PyPI. This can be handy.
>
> ----------------------------------------------------------------------
>
>>Comment By: Martin v. L?wis (loewis)
> Date: 2012-02-18 09:14
>
> Message:
> You cannot. This is intentional; the Sphinx authors chose to hide all but
> the current version from you.
>
> Closing the report as "works for me"; PyPI does support listing multiple
> versions, and the feature seems to work correctly as designed.
>
> ----------------------------------------------------------------------
>
> Comment By: anatoly techtonik (techtonik)
> Date: 2012-02-18 06:47
>
> Message:
> All right. So, how can I list all Sphinx versions available from PyPI using
> web interface of Python Package Index?
>
> ----------------------------------------------------------------------
>
> Comment By: Martin v. L?wis (loewis)
> Date: 2012-02-18 06:20
>
> Message:
> Becausse only a single releae of Sphinx is visible.
>
> ----------------------------------------------------------------------
>
> Comment By: anatoly techtonik (techtonik)
> Date: 2012-02-18 05:26
>
> Message:
> Why I can't see anything for Sphinx?
>
> http://pypi.python.org/pypi/Sphinx
>
> ----------------------------------------------------------------------
>
> Comment By: Martin v. L?wis (loewis)
> Date: 2012-02-18 04:11
>
> Message:
> It most certainly supports listing all versions. E.g. if you go to
>
> http://pypi.python.org/pypi/Django
>
> you see all Django versions.
>
> ----------------------------------------------------------------------
>
> You can respond by visiting:
> https://sourceforge.net/tracker/?func=detail&atid=513503&aid=3488989&group_id=66150
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>



-- 
'Knowledge is Power'
Daniel Greenfeld
http://pydanny.blogspot.com

From drkjam at gmail.com  Wed Feb 22 21:04:47 2012
From: drkjam at gmail.com (drkjam)
Date: Wed, 22 Feb 2012 20:04:47 +0000
Subject: [Catalog-sig] pypi.python.org configuration
In-Reply-To: <CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
Message-ID: <FA5D50FB-0603-4559-947D-5AAC303F314E@gmail.com>

Would it be possible to get hold of a sanitized version of the Apache config running pypi.python.org, in particular the Apache mod_rewrite rules?

Regards,

Dave M.

From richard at python.org  Thu Feb 23 00:27:44 2012
From: richard at python.org (Richard Jones)
Date: Thu, 23 Feb 2012 10:27:44 +1100
Subject: [Catalog-sig] pypi.python.org configuration
In-Reply-To: <FA5D50FB-0603-4559-947D-5AAC303F314E@gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
	<FA5D50FB-0603-4559-947D-5AAC303F314E@gmail.com>
Message-ID: <CAHrZfZA-8fzMNYSXeVWneQCKTx1s5b0HdcbNYNEREUHSUKMnUg@mail.gmail.com>

On 23 February 2012 07:04, drkjam <drkjam at gmail.com> wrote:
> Would it be possible to get hold of a sanitized version of the Apache config running pypi.python.org, in particular the Apache mod_rewrite rules?

PyPI doesn't run behind Apache any more; Martin moved it to
nginx/uwsgi to address Apache performance issues we were running into.

There's only one rewrite in the current configuration for pypi.python.org:

        rewrite ^/$ $scheme://pypi.python.org/pypi redirect;

Was there something specific you wanted to know?


    Richard

From drkjam at gmail.com  Thu Feb 23 08:13:49 2012
From: drkjam at gmail.com (drkjam)
Date: Thu, 23 Feb 2012 07:13:49 +0000
Subject: [Catalog-sig] pypi.python.org configuration
In-Reply-To: <CAHrZfZA-8fzMNYSXeVWneQCKTx1s5b0HdcbNYNEREUHSUKMnUg@mail.gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
	<FA5D50FB-0603-4559-947D-5AAC303F314E@gmail.com>
	<CAHrZfZA-8fzMNYSXeVWneQCKTx1s5b0HdcbNYNEREUHSUKMnUg@mail.gmail.com>
Message-ID: <D75395B1-4BDB-4792-94BB-EF87C7E1617D@gmail.com>

On 22 Feb 2012, at 23:27, Richard Jones <richard at python.org> wrote:

> On 23 February 2012 07:04, drkjam <drkjam at gmail.com> wrote:
>> Would it be possible to get hold of a sanitized version of the Apache config running pypi.python.org, in particular the Apache mod_rewrite rules?
> 
> PyPI doesn't run behind Apache any more; Martin moved it to
> nginx/uwsgi to address Apache performance issues we were running into.
> 
> There's only one rewrite in the current configuration for pypi.python.org:
> 
>        rewrite ^/$ $scheme://pypi.python.org/pypi redirect;
> 
> Was there something specific you wanted to know?
> 
> 
>    Richard

Ah, good to know, thanks.

I noticed (and replicated) the behaviour of the rewrite rule you mention. Also I've observed that URLs to packages with dashes in their names get redirected to ones where these have been replaced by underscores e.g. cx-Oracle -> cx_Oracle. I've just released a modified clone (chishop, Apache/mod_wsgi) for internal use at a company and ran into this one.

Are there any other URL mods lurking that I should be aware of to be added to my implementation?

From martin at v.loewis.de  Thu Feb 23 22:20:55 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 23 Feb 2012 22:20:55 +0100
Subject: [Catalog-sig] pypi.python.org configuration
In-Reply-To: <D75395B1-4BDB-4792-94BB-EF87C7E1617D@gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>	<FA5D50FB-0603-4559-947D-5AAC303F314E@gmail.com>	<CAHrZfZA-8fzMNYSXeVWneQCKTx1s5b0HdcbNYNEREUHSUKMnUg@mail.gmail.com>
	<D75395B1-4BDB-4792-94BB-EF87C7E1617D@gmail.com>
Message-ID: <4F46ADB7.70202@v.loewis.de>

> I noticed (and replicated) the behaviour of the rewrite rule you
> mention. Also I've observed that URLs to packages with dashes in
> their names get redirected to ones where these have been replaced by
> underscores e.g. cx-Oracle -> cx_Oracle. I've just released a
> modified clone (chishop, Apache/mod_wsgi) for internal use at a
> company and ran into this one.
> 
> Are there any other URL mods lurking that I should be aware of to be
> added to my implementation? 

PyPI does Redirects in many cases, but not in the web server. See

https://svn.python.org/packages/trunk/pypi/webui.py

Regards,
Martin

From drkjam at gmail.com  Fri Feb 24 08:11:14 2012
From: drkjam at gmail.com (drkjam)
Date: Fri, 24 Feb 2012 07:11:14 +0000
Subject: [Catalog-sig] pypi.python.org configuration
In-Reply-To: <4F46ADB7.70202@v.loewis.de>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
	<FA5D50FB-0603-4559-947D-5AAC303F314E@gmail.com>
	<CAHrZfZA-8fzMNYSXeVWneQCKTx1s5b0HdcbNYNEREUHSUKMnUg@mail.gmail.com>
	<D75395B1-4BDB-4792-94BB-EF87C7E1617D@gmail.com>
	<4F46ADB7.70202@v.loewis.de>
Message-ID: <8E7C1000-6746-4291-9DA0-88F6A2030024@gmail.com>

On 23 Feb 2012, at 21:20, "Martin v. L?wis" <martin at v.loewis.de> wrote:

>> I noticed (and replicated) the behaviour of the rewrite rule you
>> mention. Also I've observed that URLs to packages with dashes in
>> their names get redirected to ones where these have been replaced by
>> underscores e.g. cx-Oracle -> cx_Oracle. I've just released a
>> modified clone (chishop, Apache/mod_wsgi) for internal use at a
>> company and ran into this one.
>> 
>> Are there any other URL mods lurking that I should be aware of to be
>> added to my implementation? 
> 
> PyPI does Redirects in many cases, but not in the web server. See
> 
> https://svn.python.org/packages/trunk/pypi/webui.py
> 
> Regards,
> Martin

OK, I take a look through the code.

Many thanks,

Dave M.

From noah at coderanger.net  Fri Feb 24 08:21:11 2012
From: noah at coderanger.net (Noah Kantrowitz)
Date: Thu, 23 Feb 2012 23:21:11 -0800
Subject: [Catalog-sig] [Infrastructure] [Roto Rooters] dinsdale dead
In-Reply-To: <4F473375.9070504@gmx.net>
References: <CAHrZfZDGYOF1oSutyRbL6rBsmZnT7N=x3kwWYXf+x7jjFLQ_jA@mail.gmail.com>
	<4F473375.9070504@gmx.net>
Message-ID: <DE879F64-4459-437A-8586-FB191FC132A6@coderanger.net>

Postgres came back up with a corrupted WAL, preventing writes to PyPI. I have backed up the WAL and used pg_resetxfer log to clear it. This almost certainly caused some amount of data loss, but I have no idea how much (probably just the 5 minutes or so before it died). Things seem stable now though, so I'm going to bed.

--Noah

On Feb 23, 2012, at 10:51 PM, Georg Brandl wrote:

> Am 24.02.2012 06:50, schrieb Richard Jones:
>> Hi roto rooters,
>> 
>> dinsdale's an ex-server at the moment, not sure what's up.
>> 
>> If it helps I'd like to put my hand up to be able to kick it in the
>> head when it becomes completely unresponsive. I'm in the UTC+10
>> timezone when a bunch of Northern Hemisphere folk are offline. Can't
>> guarantee availability but might be able to help on occasion.
> 
> I've now powercycled the machine.
> 
> Noah, Richard, please contact MvL for access to the xs4all infrastructure.
> 
> Georg
> 
> ________________________________________________
> Infrastructure mailing list
> Infrastructure at python.org
> http://mail.python.org/mailman/listinfo/infrastructure
> Unsubscribe: http://mail.python.org/mailman/options/infrastructure/noah%40coderanger.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120223/e3a6e5dc/attachment.pgp>

From techtonik at gmail.com  Tue Feb 28 12:07:54 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 28 Feb 2012 14:07:54 +0300
Subject: [Catalog-sig] Fwd: [ pypi-Bugs-3488989 ] Need a way to list all
 package versions
In-Reply-To: <0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
	<0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>
Message-ID: <CAPkN8x+GYY=O3wAoJD678uH7jw=riQ5LL=9JiX0K1+kjWFn89Q@mail.gmail.com>

On Sun, Feb 19, 2012 at 12:07 AM, Donald Stufft <donald.stufft at gmail.com> wrote:
> On Saturday, February 18, 2012 at 4:01 PM, anatoly techtonik wrote:
>
> Hi,
>
> I find SF tracker very annoying, because it doesn't even allow to add
> comments to a closed issue (see below). That's why crossposting here.
>
> So, there two questions:
> 1. does anybody else also thinks that?having a way to list?all package
> versions available on PyPI would be awesome?
>
> It's possible via the API to list all packages regardless of hidden or not.
> The Web UI doesn't support it. I'm assuming from the ticket PyPI is happy
> with the way it works, but fwiw you can see all the package versions for
> Sphinx at?http://crate.io/packages/Sphinx/? (click the All Versions tab).

Thanks for replies about API, I used XML-RPC myself, but it's not a
good user experience to require Python shell to get information from
the web site. http://pypi.python.org/packages/source/S/Sphinx/ really
helps to see what packages are available using browser alone, but
unfortunately it is a hidden magic.

> 2. is there people who don't like SF tracker?
>
> Haven't really used the SF tracker, but if it doesn't allow comments on
> closed issues that would be mildly annoying to me.

I wonder what are the reasons not to use it?
-- 
anatoly t.

From richard at python.org  Tue Feb 28 12:35:58 2012
From: richard at python.org (Richard Jones)
Date: Tue, 28 Feb 2012 22:35:58 +1100
Subject: [Catalog-sig] Fwd: [ pypi-Bugs-3488989 ] Need a way to list all
 package versions
In-Reply-To: <CAPkN8x+GYY=O3wAoJD678uH7jw=riQ5LL=9JiX0K1+kjWFn89Q@mail.gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
	<0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>
	<CAPkN8x+GYY=O3wAoJD678uH7jw=riQ5LL=9JiX0K1+kjWFn89Q@mail.gmail.com>
Message-ID: <CAHrZfZAznj5aEkK5qEi_OPr3eeTKG_8k3DgqN1NCsyuuJUAtMQ@mail.gmail.com>

On 28 February 2012 22:07, anatoly techtonik <techtonik at gmail.com> wrote:
> On Sun, Feb 19, 2012 at 12:07 AM, Donald Stufft <donald.stufft at gmail.com> wrote:
>> On Saturday, February 18, 2012 at 4:01 PM, anatoly techtonik wrote:
>> 1. does anybody else also thinks that?having a way to list?all package
>> versions available on PyPI would be awesome?
>>
>> It's possible via the API to list all packages regardless of hidden or not.
>> The Web UI doesn't support it. I'm assuming from the ticket PyPI is happy
>> with the way it works, but fwiw you can see all the package versions for
>> Sphinx at?http://crate.io/packages/Sphinx/? (click the All Versions tab).
>
> Thanks for replies about API, I used XML-RPC myself, but it's not a
> good user experience to require Python shell to get information from
> the web site. http://pypi.python.org/packages/source/S/Sphinx/ really
> helps to see what packages are available using browser alone, but
> unfortunately it is a hidden magic.

When PyPI was created there was a design discussion and decision made
that the site implement the most common case which is that package
authors expose only one version, the most recent one. Unsupported
versions are deliberately difficult to find. That's the point. The
intention is to reduce the authors' maintenance burden by discouraging
people from using older versions of their software.

Your example project, Sphinx, advertises precisely one current
version, 1.1.2. This is the one on their website and it's the one on
PyPI.

Some packages expose more versions because they have multiple release
branches and that's their choice. More versions are findable on the
site for those packages.

And, as has been pointed out, for data mining applications you can get
access to the full data.


     Richard

From techtonik at gmail.com  Wed Feb 29 16:57:22 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 29 Feb 2012 17:57:22 +0200
Subject: [Catalog-sig] Fwd: [ pypi-Bugs-3488989 ] Need a way to list all
 package versions
In-Reply-To: <CAHrZfZAznj5aEkK5qEi_OPr3eeTKG_8k3DgqN1NCsyuuJUAtMQ@mail.gmail.com>
References: <4f3fdc92.08762b0a.4827.ffff9ba2SMTPIN_ADDED@mx.google.com>
	<CAPkN8x+DpD9_+adsME+uHeb+0R+cru3sF3cKvgLyfsdDWrfmNA@mail.gmail.com>
	<0002C2A2ABB840838DD2C1F75B8B6678@gmail.com>
	<CAPkN8x+GYY=O3wAoJD678uH7jw=riQ5LL=9JiX0K1+kjWFn89Q@mail.gmail.com>
	<CAHrZfZAznj5aEkK5qEi_OPr3eeTKG_8k3DgqN1NCsyuuJUAtMQ@mail.gmail.com>
Message-ID: <CAPkN8x+KLgNhn4+6rbTZZLemsS69aKLwZEwn_dbvPyFgb-Qk+Q@mail.gmail.com>

On Tue, Feb 28, 2012 at 2:35 PM, Richard Jones <richard at python.org> wrote:
> On 28 February 2012 22:07, anatoly techtonik <techtonik at gmail.com> wrote:
>> On Sun, Feb 19, 2012 at 12:07 AM, Donald Stufft <donald.stufft at gmail.com> wrote:
>>> On Saturday, February 18, 2012 at 4:01 PM, anatoly techtonik wrote:
>>> 1. does anybody else also thinks that?having a way to list?all package
>>> versions available on PyPI would be awesome?
>>>
>>> It's possible via the API to list all packages regardless of hidden or not.
>>> The Web UI doesn't support it. I'm assuming from the ticket PyPI is happy
>>> with the way it works, but fwiw you can see all the package versions for
>>> Sphinx at?http://crate.io/packages/Sphinx/? (click the All Versions tab).
>>
>> Thanks for replies about API, I used XML-RPC myself, but it's not a
>> good user experience to require Python shell to get information from
>> the web site. http://pypi.python.org/packages/source/S/Sphinx/ really
>> helps to see what packages are available using browser alone, but
>> unfortunately it is a hidden magic.
>
> When PyPI was created there was a design discussion and decision made
> that the site implement the most common case which is that package
> authors expose only one version, the most recent one. Unsupported
> versions are deliberately difficult to find. That's the point. The
> intention is to reduce the authors' maintenance burden by discouraging
> people from using older versions of their software.
>
> Your example project, Sphinx, advertises precisely one current
> version, 1.1.2. This is the one on their website and it's the one on
> PyPI.
>
> Some packages expose more versions because they have multiple release
> branches and that's their choice. More versions are findable on the
> site for those packages.
>
> And, as has been pointed out, for data mining applications you can get
> access to the full data.
>
>
> ? ? Richard

Thanks for the proper explanation. At last, after 11 days and
catalog-sig subscription. =) This proves that SF tracker is useless
and fails to communicate with users. It would be interesting to
glimpse at the archives of this design discussion. I can not agree
that users should be deliberately limited.
--
anatoly t.