From jannis at leidel.info  Mon Jul  2 14:11:40 2012
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 2 Jul 2012 14:11:40 +0200
Subject: [Catalog-sig] Some pypi mirrors not up to date?
In-Reply-To: <4DBF0FB4.5060900@v.loewis.de>
References: <ipml5d$l4k$1@dough.gmane.org>
	<9B68E47E-1639-445D-AF5F-3791BB5CCF7D@leidel.info>
	<4DBF0FB4.5060900@v.loewis.de>
Message-ID: <86BBA972-78A3-4D3C-A63E-FC327EF80E76@leidel.info>

On 02.05.2011, at 22:10, Martin v. L?wis <martin at v.loewis.de> wrote:

> Am 02.05.2011 19:24, schrieb Jannis Leidel:
>> 
>> On 02.05.2011, at 18:12, Maurits van Rees wrote:
>> 
>>> Hi,
>>> 
>>> I noticed that some distributions are not on all mirrors.  For example
>>> http://a.pypi.python.org/simple/plone.app.referenceablebehavior/
>>> has 0.1 and 0.2 (last one released 30 April)
>>> but 0.2 is missing from
>>> http://b.pypi.python.org/simple/plone.app.referenceablebehavior/
>>> 
>>> Same for c and d.  Ah, no: those two have it now.  I know for sure that at least d did not have it five minutes ago.  And this version has been released two days ago, so it should have been slightly faster. :-)
>> 
>> Hm, d doesn't seem to have the file on disk even thought it's on the simple page, see http://d.pypi.python.org/packages/source/p/plone.app.referenceablebehavior/
>> 
>> Martin: Anything I can do to make sure this doesn't happen again?
> 
> As the starting point, we should figure out why it happened in
> the first place - it shouldn't have, of course. Most likely,
> it's a bug :-)

Looks like http://b.pypi.python.org is out of date again: http://www.pypi-mirrors.org

Can we do something about that?

Jannis


From kencochrane at gmail.com  Mon Jul  2 14:37:20 2012
From: kencochrane at gmail.com (ken cochrane)
Date: Mon, 2 Jul 2012 08:37:20 -0400
Subject: [Catalog-sig] Some pypi mirrors not up to date?
In-Reply-To: <86BBA972-78A3-4D3C-A63E-FC327EF80E76@leidel.info>
References: <ipml5d$l4k$1@dough.gmane.org>
	<9B68E47E-1639-445D-AF5F-3791BB5CCF7D@leidel.info>
	<4DBF0FB4.5060900@v.loewis.de>
	<86BBA972-78A3-4D3C-A63E-FC327EF80E76@leidel.info>
Message-ID: <CACBKDngEq8y5yFFPYo-Ymwd8f655tUviMXkw=JdheJhrfkW1Lg@mail.gmail.com>

Jannis,

I started looking at this the other day, and I haven't had a chance to fix
it because the Amazon datacenter outage took all of my time the past few
days.

Here is what I found out.

b.pypi.python.org lives on GAE and it's currently stuck, and looking at the
logs I figured out what went wrong, but I'm not sure how to fix it. [3] See
log snippet at the end of the email.

Basically there is a python package called '__past__'  (see [0] link below)
that is causing the sync process to break because we are trying to use the
project name as the key_name for the Product model [1], and GAE model
key_name's can't contain underscores [2].

I'm not sure how to fix the issue without possibly breaking other things.
My first thought was to remove the underscores, but that might break
something else, or conflict with another project with a similar name. I
wrote to Martin who gave me the following advice.

>From "Martin v. L?wis":

Renaming/escaping sounds good. I'd check if there is any string that
> can be used in a GAE key name, but not be used in a PyPI package name.
> If not, standard escaping needs to be applied: a prefix of "dunder"
> is added to any package whose name starts and ends with __, as well
> as to any package whose name starts with "dunder".
>


> When looking at all child nodes, remove "dunder" from any name;
> when doing lookups by name, escape as above.
>


> If you do find a character/string that can be in a key name but
> not in a package name, just escape the string with that name -
> no need to worry about escaping the escape character. However it
> may be that the only possible choice is "/" (which I know cannot
> appear in a package name).


I looked through most of the pypi code and I think the only character you
can't have is "/", all other characters look like they work.

So, I know what is causing it, we just need to fix the issue, test it, and
roll out the fix. I was planning on doing it this past weekend but thanks
to AWS, I didn't have any time to work on it. If anyone has any free time,
feel free to take over / help. Just let others know so there isn't any
duplicate effort.

Let me know if you have any questions.

Ken Cochrane

Footnotes:

[0] http://pypi.python.org/pypi/__past__/0.0.1.dev

[1]
https://bitbucket.org/loewis/pypi-appengine/src/fa6596a427e1/fetch.py#cl-62


[2] Information about model key_names
https://developers.google.com/appengine/docs/python/datastore/modelclass#Model
key_name

The key name for the entity. The name becomes part of the primary key. If
None, a system-generated numeric ID is used for the key.

The value for key_name must not be of the form __*__.
[1]
Log snippet.

   1. 2012-06-28 06:45:18.222

   step package '__past__'

   2. E2012-06-28 06:45:18.778

   illegal name in key path element: __past__
   Traceback (most recent call last):
     File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py",
line 701, in __call__
       handler.get(*groups)
     File "/base/data/home/apps/pypi/3.358089379617981219/handlers.py",
line 171, in get
       self.response.out.write(fetch.cron())
     File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py",
line 293, in cron
       return step()
     File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py",
line 259, in step
       actions[action](m, todo, param)
     File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py",
line 91, in package
       data = simple_page(m, name)
     File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py",
line 70, in simple_page
       obj.put()
     File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py",
line 1074, in put
       return datastore.Put(self._entity, **kwargs)
     File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py",
line 579, in Put
       return PutAsync(entities, **kwargs).get_result()
     File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
line 604, in get_result
       return self.__get_result_hook(self)
     File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
line 1579, in __put_hook
       self.check_rpc_success(rpc)
     File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
line 1216, in check_rpc_success
       raise _ToDatastoreError(err)
   BadRequestError: illegal name in key path element: __past__




On Mon, Jul 2, 2012 at 8:11 AM, Jannis Leidel <jannis at leidel.info> wrote:

> On 02.05.2011, at 22:10, Martin v. L?wis <martin at v.loewis.de> wrote:
>
> > Am 02.05.2011 19:24, schrieb Jannis Leidel:
> >>
> >> On 02.05.2011, at 18:12, Maurits van Rees wrote:
> >>
> >>> Hi,
> >>>
> >>> I noticed that some distributions are not on all mirrors.  For example
> >>> http://a.pypi.python.org/simple/plone.app.referenceablebehavior/
> >>> has 0.1 and 0.2 (last one released 30 April)
> >>> but 0.2 is missing from
> >>> http://b.pypi.python.org/simple/plone.app.referenceablebehavior/
> >>>
> >>> Same for c and d.  Ah, no: those two have it now.  I know for sure
> that at least d did not have it five minutes ago.  And this version has
> been released two days ago, so it should have been slightly faster. :-)
> >>
> >> Hm, d doesn't seem to have the file on disk even thought it's on the
> simple page, see
> http://d.pypi.python.org/packages/source/p/plone.app.referenceablebehavior/
> >>
> >> Martin: Anything I can do to make sure this doesn't happen again?
> >
> > As the starting point, we should figure out why it happened in
> > the first place - it shouldn't have, of course. Most likely,
> > it's a bug :-)
>
> Looks like http://b.pypi.python.org is out of date again:
> http://www.pypi-mirrors.org
>
> Can we do something about that?
>
> Jannis
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120702/d94d932d/attachment-0001.html>

From richard at python.org  Thu Jul  5 04:18:32 2012
From: richard at python.org (Richard Jones)
Date: Thu, 5 Jul 2012 12:18:32 +1000
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <5256255447244850384@unknownmsgid>
References: <5256255447244850384@unknownmsgid>
Message-ID: <CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>

On 23 June 2012 10:21, Aaron Meurer <asmeurer at gmail.com> wrote:
> There's also the issue that every
> time we put out a release candidate for a new version, pip starts
> installing that, when I would prefer it to only install stable final
> releases.  It's also, as I noted on the other discussion list, a bit
> of a security risk.
>
> According to the pip guys (namely, Carl Meyer), this is not so easy to
> change from their end because of backwards compatibility issues.  I
> suggested that such a flag be added to PyPI, and they told me that if
> it were, they would accept a patch supporting it in pip.  This would
> make it much less of a headache for me as a package maintainer,
> because I could know that pip will always install exactly what I want.
> It could be off by default to enable backwards compatibility.

Just to be clear, what's being proposed is some way to flag a release
on PyPI as being "stable" (or some other release as "unstable")? Then
a tool such as pip could prefer a stable release over an unstable
release while scraping download links from PyPI and all related pages?
How would this flag be presented to pip? How would a package
maintainer manage it?

Just a thought on the version number precedence rules: would using
"smypy-rc1-0.7.1" rather than "sympy-0.7.1-rc1" (ie. your version is
'rc1-0.7.1" instead of "0.7.1-rc1") work? It's a hack, I know, but I
had another look at the easy_install docs page and it's not clear to
me whether that would work. I think it might because it uses the same
basic work-around as the sympy-docs-html file.


     Richard

From asmeurer at gmail.com  Thu Jul  5 06:43:39 2012
From: asmeurer at gmail.com (Aaron Meurer)
Date: Wed, 4 Jul 2012 22:43:39 -0600
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
Message-ID: <-8494424675407955308@unknownmsgid>

On Jul 4, 2012, at 8:18 PM, Richard Jones <richard at python.org> wrote:

> On 23 June 2012 10:21, Aaron Meurer <asmeurer at gmail.com> wrote:
>> There's also the issue that every
>> time we put out a release candidate for a new version, pip starts
>> installing that, when I would prefer it to only install stable final
>> releases.  It's also, as I noted on the other discussion list, a bit
>> of a security risk.
>>
>> According to the pip guys (namely, Carl Meyer), this is not so easy to
>> change from their end because of backwards compatibility issues.  I
>> suggested that such a flag be added to PyPI, and they told me that if
>> it were, they would accept a patch supporting it in pip.  This would
>> make it much less of a headache for me as a package maintainer,
>> because I could know that pip will always install exactly what I want.
>> It could be off by default to enable backwards compatibility.
>
> Just to be clear, what's being proposed is some way to flag a release
> on PyPI as being "stable" (or some other release as "unstable")? Then
> a tool such as pip could prefer a stable release over an unstable
> release while scraping download links from PyPI and all related pages?
> How would this flag be presented to pip? How would a package
> maintainer manage it?

I think the cleanest way would be to just have a way to tell pip to
only install the files that are uploaded to PyPI (alternately, files
from a direct download link).  In other words, I want to force
pip/easy_install to *not* do any link scraping.

The way I visualize it is somewhere in the PyPI package page, there is
a checkbox, off by default, that says something like, "PyPI only." and
the descriptor text would be something like, "This will prevent tools
like pip and easy_install from link scraping to find the most recent
version of this package.  Only the uploaded file for the most recent
version on PyPI will be downloaded by these tools, even if a newer
version might be found on another website.  This is useful if you want
to prevent pip/easy_install from downloading incorrect files that it
thinks are newer, or if you want it to only download a stable version,
but remember that if you check this, it is up to you to update the
package here on PyPI when new versions are released or
pip/easy_install will never install them."

I think would have to be a package-wide setting (unlike most settings,
which are version-wide) because you're telling it to always use the
newest version on PyPI, regardless of what that is.

>
> Just a thought on the version number precedence rules: would using
> "smypy-rc1-0.7.1" rather than "sympy-0.7.1-rc1" (ie. your version is
> 'rc1-0.7.1" instead of "0.7.1-rc1") work? It's a hack, I know, but I
> had another look at the easy_install docs page and it's not clear to
> me whether that would work. I think it might because it uses the same
> basic work-around as the sympy-docs-html file.

Yes, I know I could hack my way to it, but I'd really like to be able
to just check a box and forget about it.

And by the way, 0.7.1-rc1 actually *is* newer than 0.7.0.  Another
possibility if this flag is implemented is an optional flag to pip
that would tell it to ignore it, for those who really do want the
newest version (and know the risks).  I have no qualms against people
being able to pip install release candidates, I just don't want it to
happen by default.

Aaron Meurer

>
>
>     Richard

From donald.stufft at gmail.com  Thu Jul  5 07:00:35 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Thu, 5 Jul 2012 01:00:35 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <-8494424675407955308@unknownmsgid>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
Message-ID: <1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>

On Thursday, July 5, 2012 at 12:43 AM, Aaron Meurer wrote:
> I think the cleanest way would be to just have a way to tell pip to
> only install the files that are uploaded to PyPI (alternately, files
> from a direct download link). In other words, I want to force
> pip/easy_install to *not* do any link scraping.
> 
> 

Sounds like something that honestly belongs in pip.

Something like ``pip --disable-external``.

Possibly something like ``pip --only-stable`` or something (if versions can be parsed by PEP 345?).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/b0a6da73/attachment.html>

From richard at python.org  Thu Jul  5 07:20:24 2012
From: richard at python.org (Richard Jones)
Date: Thu, 5 Jul 2012 15:20:24 +1000
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
Message-ID: <CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>

On 5 July 2012 15:00, Donald Stufft <donald.stufft at gmail.com> wrote:
> I think the cleanest way would be to just have a way to tell pip to
> only install the files that are uploaded to PyPI (alternately, files
> from a direct download link). In other words, I want to force
> pip/easy_install to *not* do any link scraping.
>
> Sounds like something that honestly belongs in pip.
>
> Something like ``pip --disable-external``.
>
> Possibly something like ``pip --only-stable`` or something (if versions can
> be parsed by PEP 345?).

This sort of thing would be nice to have - especially if it could be
configured to be on by default (with command-line args to enable the
reverse behaviour.) Though to be quite honest I don't actually know
how many packages I currently use that aren't downloaded directly from
the cheeseshop.

I wonder whether we could have a "stable" index view on PyPI which
displays the above, just like your "restricted" view? Just thinking
out loud - I don't actually have any time to implement such a thing
right now :-)


    Richard

From donald.stufft at gmail.com  Thu Jul  5 07:26:59 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Thu, 5 Jul 2012 01:26:59 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
Message-ID: <91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>

On Thursday, July 5, 2012 at 1:20 AM, Richard Jones wrote:
> This sort of thing would be nice to have - especially if it could be
> configured to be on by default (with command-line args to enable the
> reverse behaviour.) Though to be quite honest I don't actually know
> how many packages I currently use that aren't downloaded directly from
> the cheeseshop.

I was unpleasantly surprised by a few that I use that aren't directly from the index. 

I don't know how likely an on by default option would be as that would be a fairly
major change and will break for a lot of people who don't know to use it. I would
be majorly for it being on by default otherwise.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/a34db534/attachment-0001.html>

From richard at python.org  Thu Jul  5 08:13:40 2012
From: richard at python.org (Richard Jones)
Date: Thu, 5 Jul 2012 16:13:40 +1000
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
Message-ID: <CAHrZfZBDBzhLaz5hfUdv0xN0tnLgQZSUezhHLP=9_A599p0F8w@mail.gmail.com>

On 5 July 2012 15:26, Donald Stufft <donald.stufft at gmail.com> wrote:
> On Thursday, July 5, 2012 at 1:20 AM, Richard Jones wrote:
>
> This sort of thing would be nice to have - especially if it could be
> configured to be on by default (with command-line args to enable the
> reverse behaviour.) Though to be quite honest I don't actually know
> how many packages I currently use that aren't downloaded directly from
> the cheeseshop.
>
> I was unpleasantly surprised by a few that I use that aren't directly from
> the index.
>
> I don't know how likely an on by default option would be as that would be a
> fairly
> major change and will break for a lot of people who don't know to use it. I
> would
> be majorly for it being on by default otherwise.

Just to be clear I'm not arguing for this to be the default behaviour
for all users of pip - just that for me I'd like to be able to
configure it to be on.


    Richard

From donald.stufft at gmail.com  Thu Jul  5 08:14:42 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Thu, 5 Jul 2012 02:14:42 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <CAHrZfZBDBzhLaz5hfUdv0xN0tnLgQZSUezhHLP=9_A599p0F8w@mail.gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<CAHrZfZBDBzhLaz5hfUdv0xN0tnLgQZSUezhHLP=9_A599p0F8w@mail.gmail.com>
Message-ID: <2786EA3653674F5FB6570489FA0A7FE8@gmail.com>

On Thursday, July 5, 2012 at 2:13 AM, Richard Jones wrote:
> Just to be clear I'm not arguing for this to be the default behaviour
> for all users of pip - just that for me I'd like to be able to
> configure it to be on.
> 
> 

Gotcha ;) I wish it could be default for all users of pip but I doubt the 
likelihood of that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/881efe8c/attachment.html>

From martin at v.loewis.de  Thu Jul  5 08:31:19 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Thu, 05 Jul 2012 08:31:19 +0200
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
Message-ID: <20120705083119.Horde.-p_XaLuWis5P9TS3P7XHi-A@webmail.df.eu>

> How would this flag be presented to pip? How would a package
> maintainer manage it?

One way of presenting it would be a separate index URL, such
as pypi.python.org/unstable/. Releases would have to be marked
as unstable during upload, and then appear only in the unstable
index (and in the owner's release management page).

Tools that download wouldn't need any extra configuration except
being pointed to the alternate index.

Regards,
Martin



From stefan-usenet at bytereef.org  Thu Jul  5 08:44:22 2012
From: stefan-usenet at bytereef.org (Stefan Krah)
Date: Thu, 5 Jul 2012 08:44:22 +0200
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
Message-ID: <20120705064422.GA22942@sleipnir.bytereef.org>

Donald Stufft <donald.stufft at gmail.com> wrote:
>     This sort of thing would be nice to have - especially if it could be
>     configured to be on by default (with command-line args to enable the
>     reverse behaviour.) Though to be quite honest I don't actually know
>     how many packages I currently use that aren't downloaded directly from
>     the cheeseshop.
> 
> I was unpleasantly surprised by a few that I use that aren't directly from the
> index.

And many people have been pleasantly surprised by external packages.

And package managers like http://code.activestate.com/pypm/ don't have
any problems with external packages. They even automatically generate
binaries.


> I don't know how likely an on by default option would be as that would be a
> fairly major change and will break for a lot of people who don't know to use it.

Indeed.


Stefan Krah



From donald.stufft at gmail.com  Thu Jul  5 09:08:51 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Thu, 5 Jul 2012 03:08:51 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <20120705064422.GA22942@sleipnir.bytereef.org>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<20120705064422.GA22942@sleipnir.bytereef.org>
Message-ID: <10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>

On Thursday, July 5, 2012 at 2:44 AM, Stefan Krah wrote:
> And many people have been pleasantly surprised by external packages.
> 
> 

I can't imagine a situation where i'd want an external package over one 
hosted on PyPI. Out of curiosity what benefits are those people
seeing from them? The only thing I can think of is for projects
where PyPI doesn't allow them to upload because their distributions
are too large (PySide I think?). Otherwise all the other properties
of external packages lead themselves to surprising behavior, higher
likelihood that any particular set of requirements will not be available,
and increase the surface for someone to compromise and exploit people
installing via pip/easy_install via PyPI.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/32b02428/attachment.html>

From asmeurer at gmail.com  Thu Jul  5 09:50:44 2012
From: asmeurer at gmail.com (Aaron Meurer)
Date: Thu, 5 Jul 2012 01:50:44 -0600
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<20120705064422.GA22942@sleipnir.bytereef.org>
	<10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>
Message-ID: <-2394317826911798556@unknownmsgid>

On Jul 5, 2012, at 1:09 AM, Donald Stufft <donald.stufft at gmail.com> wrote:

On Thursday, July 5, 2012 at 2:44 AM, Stefan Krah wrote:

And many people have been pleasantly surprised by external packages.

 I can't imagine a situation where i'd want an external package over one
hosted on PyPI. Out of curiosity what benefits are those people
seeing from them? The only thing I can think of is for projects
where PyPI doesn't allow them to upload because their distributions
are too large (PySide I think?).


I think the other potential reason Carl mentioned was legal reasons.  I
have no idea what those might be, though.

Personally, if I had to guess, most packages that aren't uploaded to PyPI
are simply due to laziness of the maintainer, coupled with the fact that
because of the searching algorithms in pip/easy_install, they really don't
have to.  IMHO, if maintainers want their packages to be pip installable,
then it's quite reasonable to expect them to keep PyPI up to date.

And note that part of my suggestion is to allow direct download links, so
if uploading is a problem for whatever reason, it should not hinder access.

Otherwise all the other properties
of external packages lead themselves to surprising behavior, higher
likelihood that any particular set of requirements will not be available,
and increase the surface for someone to compromise and exploit people
installing via pip/easy_install via PyPI.


Thanks, I think this summarizes the situation nicely.

By the way, I'm curious just how many packages a change in policy would
affect.  How many packages don't have uploads?  How many packages is pip
installing a version newer than the most recent one listed on PyPI?  My
guess is that the numbers would be quite high.

Aaron Meurer

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG at python.org
http://mail.python.org/mailman/listinfo/catalog-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/01ac286a/attachment-0001.html>

From tarek at ziade.org  Thu Jul  5 09:53:04 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 05 Jul 2012 09:53:04 +0200
Subject: [Catalog-sig] Feature idea for PyPI third-party projects
Message-ID: <4FF547E0.4050000@ziade.org>

Hey,

Maybe this is already done somewhere, but here's a feature I need :

I'd like to 'watch' some projects and get an e-mail every time they 
release a new version at PyPI.


Cheers
Tarek

From donald.stufft at gmail.com  Thu Jul  5 10:06:11 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Thu, 5 Jul 2012 04:06:11 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <-2394317826911798556@unknownmsgid>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<20120705064422.GA22942@sleipnir.bytereef.org>
	<10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>
	<-2394317826911798556@unknownmsgid>
Message-ID: <111B0FB9412E463C8D5154EE713A3C18@gmail.com>

On Thursday, July 5, 2012 at 3:50 AM, Aaron Meurer wrote:
> By the way, I'm curious just how many packages a change in policy would affect.  How many packages don't have uploads?  How many packages is pip installing a version newer than the most recent one listed on PyPI?  My guess is that the numbers would be quite high.
> 
> 
> 
> 

Going by the current numbers on Crate.io

14,418 of 104,823 (14%) releases have no files associated with them.
5719 of 22187 (26%) projects have no files associated with them.

I don't have any easy way to determine how many of them are placeholder
or junk packages nor to determine what version pip would install (would it
be external or internal).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/2abc0c8d/attachment.html>

From donald.stufft at gmail.com  Thu Jul  5 10:07:25 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Thu, 5 Jul 2012 04:07:25 -0400
Subject: [Catalog-sig] Feature idea for PyPI third-party projects
In-Reply-To: <4FF547E0.4050000@ziade.org>
References: <4FF547E0.4050000@ziade.org>
Message-ID: <B87A41C8870F449B8AE00C52055B92B3@gmail.com>

On Thursday, July 5, 2012 at 3:53 AM, Tarek Ziad? wrote:
> I'd like to 'watch' some projects and get an e-mail every time they  
> release a new version at PyPI.
>  
>  

Hey, mentioned on twitter but figured i'd say so here as well. This is
something I plan on adding to Crate.  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/a6f8ee4d/attachment.html>

From carl at oddbird.net  Thu Jul  5 21:22:54 2012
From: carl at oddbird.net (Carl Meyer)
Date: Thu, 05 Jul 2012 13:22:54 -0600
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
Message-ID: <4FF5E98E.3030408@oddbird.net>

On 07/04/2012 11:00 PM, Donald Stufft wrote:
> On Thursday, July 5, 2012 at 12:43 AM, Aaron Meurer wrote:
>> I think the cleanest way would be to just have a way to tell pip to
>> only install the files that are uploaded to PyPI (alternately, files
>> from a direct download link). In other words, I want to force
>> pip/easy_install to *not* do any link scraping.
> Sounds like something that honestly belongs in pip.
> 
> Something like ``pip --disable-external``.
> 
> Possibly something like ``pip --only-stable`` or something (if versions
> can be parsed by PEP 345?).

I don't have any objection to a flag in pip to disable crawling off the
index domain (this is a hard security requirement for some users, and
something pip ought to have), but it doesn't at all meet Aaron's desire
as a package maintainer to be able to make this happen *by default* for
everyone pip-installing his package.

Carl

From martin at v.loewis.de  Thu Jul  5 21:50:29 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 05 Jul 2012 21:50:29 +0200
Subject: [Catalog-sig] New hostname for SSH uploads
In-Reply-To: <E4C61C83-3546-4439-8AE2-9F9F1FBEFF5D@coderanger.net>
References: <E4C61C83-3546-4439-8AE2-9F9F1FBEFF5D@coderanger.net>
Message-ID: <4FF5F005.8090804@v.loewis.de>

On 28.05.2012 09:43, Noah Kantrowitz wrote:
> Can someone with commit access please update the pypissh package
> to use the hostname "ssh.pypi.python.org".

Done with 1.4.

> What do people think is a reasonable deprecation timeline for this?
> To pull numbers out of nowhere, three months seems like a
> reasonable threshold.

Sounds fine. I suppose this will all take longer than expected anyway,
plus no matter what the deprecation time is, it will still break
for some people (unless the CDN supports proxying ssh to the real
host, which I think would technically be possible).

Regards,
Martin


From asmeurer at gmail.com  Thu Jul  5 22:06:02 2012
From: asmeurer at gmail.com (Aaron Meurer)
Date: Thu, 5 Jul 2012 14:06:02 -0600
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <4FF5E98E.3030408@oddbird.net>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<4FF5E98E.3030408@oddbird.net>
Message-ID: <CAKgW=6L2a2siKW0qGwsiJDHwr_pau7ctp=boJLOuJhmbo1C8qA@mail.gmail.com>

On Thu, Jul 5, 2012 at 1:22 PM, Carl Meyer <carl at oddbird.net> wrote:
> On 07/04/2012 11:00 PM, Donald Stufft wrote:
>> On Thursday, July 5, 2012 at 12:43 AM, Aaron Meurer wrote:
>>> I think the cleanest way would be to just have a way to tell pip to
>>> only install the files that are uploaded to PyPI (alternately, files
>>> from a direct download link). In other words, I want to force
>>> pip/easy_install to *not* do any link scraping.
>> Sounds like something that honestly belongs in pip.
>>
>> Something like ``pip --disable-external``.
>>
>> Possibly something like ``pip --only-stable`` or something (if versions
>> can be parsed by PEP 345?).
>
> I don't have any objection to a flag in pip to disable crawling off the
> index domain (this is a hard security requirement for some users, and
> something pip ought to have), but it doesn't at all meet Aaron's desire
> as a package maintainer to be able to make this happen *by default* for
> everyone pip-installing his package.
>
> Carl

Exactly.  This would be basically a waste of time if it weren't
default, because users can already get this behavior by passing
sufficient flags to pip (it might be slightly more convenient, but
that's it).  My point of view is that of a package maintainer, not a
user, who wants the user experience of people trying to install my
package using pip to be as seamless as possible.

My proposal recognizes that changing the defaults for everyone would
be a nightmare that the community is probably not ready for, but still
empowers package maintainers like myself to do so if want so.  In
other words, it would make my life easier, and it wouldn't make your
lives any harder.

Aaron Meurer

From tarek at ziade.org  Fri Jul  6 00:19:17 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 06 Jul 2012 00:19:17 +0200
Subject: [Catalog-sig] Feature idea for PyPI third-party projects
In-Reply-To: <B87A41C8870F449B8AE00C52055B92B3@gmail.com>
References: <4FF547E0.4050000@ziade.org>
	<B87A41C8870F449B8AE00C52055B92B3@gmail.com>
Message-ID: <4FF612E5.2000909@ziade.org>

On 7/5/12 10:07 AM, Donald Stufft wrote:
> On Thursday, July 5, 2012 at 3:53 AM, Tarek Ziad? wrote:
>> I'd like to 'watch' some projects and get an e-mail every time they
>> release a new version at PyPI.
> Hey, mentioned on twitter but figured i'd say so here as well. This is
> something I plan on adding to Crate.

Cool!
Markus mentioned he'd rather have a rss feed

So maybe being able to do a project selection, then create a custom feed 
and/or a mail per publication

>
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120706/2d15323a/attachment.html>

From tjreedy at udel.edu  Fri Jul  6 00:38:01 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 05 Jul 2012 18:38:01 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <-2394317826911798556@unknownmsgid>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<20120705064422.GA22942@sleipnir.bytereef.org>
	<10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>
	<-2394317826911798556@unknownmsgid>
Message-ID: <jt550i$2ac$1@dough.gmane.org>

On 7/5/2012 3:50 AM, Aaron Meurer wrote:

> I think the other potential reason Carl mentioned was legal reasons.  I
> have no idea what those might be, though.

Last I knew, uploading a file required licensing it to the PSF. On the 
other hand, I can find no mention of that on
http://pypi.python.org/pypi?%3Aaction=submit_form
http://wiki.python.org/moin/CheeseShopTutorial
nor a link to the license anywhere. So I don't know what the current 
situation is.

-- 
Terry Jan Reedy




From martin at v.loewis.de  Fri Jul  6 00:47:38 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Fri, 06 Jul 2012 00:47:38 +0200
Subject: [Catalog-sig] Feature idea for PyPI third-party projects
In-Reply-To: <4FF612E5.2000909@ziade.org>
References: <4FF547E0.4050000@ziade.org>
	<B87A41C8870F449B8AE00C52055B92B3@gmail.com>
	<4FF612E5.2000909@ziade.org>
Message-ID: <20120706004738.Horde.GqqLL8L8999P9hmKn4iWvLA@webmail.df.eu>

> Markus mentioned he'd rather have a rss feed
>
> So maybe being able to do a project selection, then create a custom  
> feed and/or a mail per publication

I'd like to point out that PyPI support pubsubhubbub. So any of these
services can be built on to of that.

Regards,
Martin



From donald.stufft at gmail.com  Fri Jul  6 00:48:09 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Thu, 5 Jul 2012 18:48:09 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <jt550i$2ac$1@dough.gmane.org>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<20120705064422.GA22942@sleipnir.bytereef.org>
	<10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>
	<-2394317826911798556@unknownmsgid> <jt550i$2ac$1@dough.gmane.org>
Message-ID: <98C61DA1FC854FDCBE0EDC0875D9A725@gmail.com>

On Thursday, July 5, 2012 at 6:38 PM, Terry Reedy wrote:
> Last I knew, uploading a file required licensing it to the PSF. On the 
> other hand, I can find no mention of that on
> http://pypi.python.org/pypi?%3Aaction=submit_form
> http://wiki.python.org/moin/CheeseShopTutorial
> nor a link to the license anywhere. So I don't know what the current 
> situation is.
> 
http://www.python.org/about/legal/

 
IANAL but I think it pretty much it just says the things you upload to the site, the site is allowed to let others download it and you don't get to charge the PSF for it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/0d65413a/attachment-0001.html>

From donald.stufft at gmail.com  Fri Jul  6 00:50:55 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Thu, 5 Jul 2012 18:50:55 -0400
Subject: [Catalog-sig] Feature idea for PyPI third-party projects
In-Reply-To: <20120706004738.Horde.GqqLL8L8999P9hmKn4iWvLA@webmail.df.eu>
References: <4FF547E0.4050000@ziade.org>
	<B87A41C8870F449B8AE00C52055B92B3@gmail.com>
	<4FF612E5.2000909@ziade.org>
	<20120706004738.Horde.GqqLL8L8999P9hmKn4iWvLA@webmail.df.eu>
Message-ID: <09887482C1B04D82894EFD3FC600530B@gmail.com>

On Thursday, July 5, 2012 at 6:47 PM, martin at v.loewis.de wrote:
> > Markus mentioned he'd rather have a rss feed
> > 
> > So maybe being able to do a project selection, then create a custom 
> > feed and/or a mail per publication
> > 
> 
> 
> I'd like to point out that PyPI support pubsubhubbub. So any of these
> services can be built on to of that.
> 
> 

I had forgotten about pubsubhubbub, I need to look at that again to
try and lessen the lag time in replication more. Do you happen to have
a link for it? 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120705/4326ac33/attachment.html>

From tjreedy at udel.edu  Fri Jul  6 04:49:58 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 05 Jul 2012 22:49:58 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <98C61DA1FC854FDCBE0EDC0875D9A725@gmail.com>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<20120705064422.GA22942@sleipnir.bytereef.org>
	<10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>
	<-2394317826911798556@unknownmsgid> <jt550i$2ac$1@dough.gmane.org>
	<98C61DA1FC854FDCBE0EDC0875D9A725@gmail.com>
Message-ID: <jt5jov$2d0$1@dough.gmane.org>

On 7/5/2012 6:48 PM, Donald Stufft wrote:
> On Thursday, July 5, 2012 at 6:38 PM, Terry Reedy wrote:
>> Last I knew, uploading a file required licensing it to the PSF. On the
>> other hand, I can find no mention of that on
>> http://pypi.python.org/pypi?%3Aaction=submit_form
>> http://wiki.python.org/moin/CheeseShopTutorial
>> nor a link to the license anywhere. So I don't know what the current
>> situation is.
>>
> http://www.python.org/about/legal/
>
> IANAL but I think it pretty much it just says the things you upload to
> the site, the site is allowed to let others download it and you don't
> get to charge the PSF for it.

That is pretty much what the first version says. The current version is 
much more expansive and seems to deny any license restrictions.

"The Python Software Foundation ... has no obligation of any kind with 
respect to such third party content."

not even to respect its license?

... "The PSF is free to use or disseminate such content on an 
unrestricted basis for any purpose,"

The purpose of any license to to restrict use or dissemination.

"and third party content providers grant the PSF and all other users of 
the web site an irrevocable, worldwide, royalty-free, nonexclusive 
license to reproduce, distribute, transmit, display, perform, and 
publish such content, including in digital form."

That says to me that the PSF *and* its users are pretty much free of any 
license restrictions on uploaded software, which negates the  point of 
having a license. For instance, without a declaration otherwise from the 
FSF, I would not think it legal to upload a derivative of a GPL-licenced 
work

---
Terry Jan Reedy




From tseaver at palladion.com  Fri Jul  6 16:19:54 2012
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 06 Jul 2012 10:19:54 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <jt5jov$2d0$1@dough.gmane.org>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<20120705064422.GA22942@sleipnir.bytereef.org>
	<10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>
	<-2394317826911798556@unknownmsgid> <jt550i$2ac$1@dough.gmane.org>
	<98C61DA1FC854FDCBE0EDC0875D9A725@gmail.com>
	<jt5jov$2d0$1@dough.gmane.org>
Message-ID: <jt6s6a$qa3$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/05/2012 10:49 PM, Terry Reedy wrote:
> On 7/5/2012 6:48 PM, Donald Stufft wrote:
>> On Thursday, July 5, 2012 at 6:38 PM, Terry Reedy wrote:
>>> Last I knew, uploading a file required licensing it to the PSF. On
>>> the other hand, I can find no mention of that on 
>>> http://pypi.python.org/pypi?%3Aaction=submit_form 
>>> http://wiki.python.org/moin/CheeseShopTutorial nor a link to the
>>> license anywhere. So I don't know what the current situation is.
>>> 
>> http://www.python.org/about/legal/
>> 
>> IANAL but I think it pretty much it just says the things you upload
>> to the site, the site is allowed to let others download it and you
>> don't get to charge the PSF for it.
> 
> That is pretty much what the first version says. The current version
> is much more expansive and seems to deny any license restrictions.
> 
> "The Python Software Foundation ... has no obligation of any kind with
>  respect to such third party content."
> 
> not even to respect its license?
> 
> ... "The PSF is free to use or disseminate such content on an 
> unrestricted basis for any purpose,"
> 
> The purpose of any license to to restrict use or dissemination.y would
> do so is

FWIW, the purpose of a copyright license is to *allow* copying which
would otherwise be prohibited by copyright.

> "and third party content providers grant the PSF and all other users
> of the web site an irrevocable, worldwide, royalty-free, nonexclusive
>  license to reproduce, distribute, transmit, display, perform, and 
> publish such content, including in digital form."
> 
> That says to me that the PSF *and* its users are pretty much free of
> any license restrictions on uploaded software, which negates the
> point of having a license.

Assuming your reading is correct, it implies that software uploaded to
PyPI itself is licensed *separately* under those terms, just as MySQL is
/ was available under both the terms of the GPL *and* a commercial
license (for those who didn't wish to abide by the GPL's terms).  Only
the copyright holder can offer such a dual license, of course.

> For instance, without a declaration otherwise from the FSF, I would
> not think it legal to upload a derivative of a GPL-licenced work

Note that the FSF's opinion is not more weighty than the author's here,
should the issue come to court:  the controlling intent would be that of
the author of the GPL'ed package from which the uploaded-to-PyPI
distribution derived.  E.g., assume that Sally releases a GPL'ed
distribution, 'foo-0.1.tar.gz', but does *not* upload to PyPI.  If Beth
creates another distribution, 'bar-0.1.zip', derived from
'foo-0.1.zip'[1], and uploads it to PyPI, then users of PyPI would have
additional permissions w.r.t. 'bar-0.1.zip', but *only to the extent that
Beth has the right under copyright law to grant them*.

[1] The example presumes that 'bar-0.1.zip' is determined under law to be
a derived work of Sally's 'foo-0.1.tar.gz'.  Whether uses which merely
import from modules in 'foo' (rather than copying) make 'bar-0.1.zip' a
derived work of 'foo-0.1.tar.gz' is a question which courts would have to
decide (the issue can reasonably be argued both ways).

That '/about/legal' language is likely drafted to cover non-software
content as well.  Perhaps it needs a clarification that uploaded software
distributions remain licensed to the site's users under the terms of any
license included in the distribution, and that permission to disstribute
it freely from PyPI is granted to the PSF as a separate license.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/29AUACgkQ+gerLs4ltQ5w+QCfU0M6q5RDmErvNd9ZY1fIq1EJ
3poAnjmLtapWGz7Tj7G18kqy1RCFRBwr
=12p6
-----END PGP SIGNATURE-----


From donald.stufft at gmail.com  Fri Jul  6 22:36:58 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Fri, 6 Jul 2012 16:36:58 -0400
Subject: [Catalog-sig] Flag to tell pip to only install uploaded files
In-Reply-To: <jt5jov$2d0$1@dough.gmane.org>
References: <5256255447244850384@unknownmsgid>
	<CAHrZfZC3pvudytWy0amr5iZOR9M9d14cnQk4sEXrT0OL3F7kfg@mail.gmail.com>
	<-8494424675407955308@unknownmsgid>
	<1FB5BD6D095F4FFFABE4D77236ABB6D6@gmail.com>
	<CAHrZfZDA9YcEdqXeKEqbCgndj09syDbe7UD6LDJyNS8yJ8RW5A@mail.gmail.com>
	<91FCEA6B2D7B4BFD846F40254D1A325C@gmail.com>
	<20120705064422.GA22942@sleipnir.bytereef.org>
	<10DEFC19C2D641C48565FEB4384A7BDA@gmail.com>
	<-2394317826911798556@unknownmsgid> <jt550i$2ac$1@dough.gmane.org>
	<98C61DA1FC854FDCBE0EDC0875D9A725@gmail.com>
	<jt5jov$2d0$1@dough.gmane.org>
Message-ID: <6044FDAAD8184B32869EB692330A854F@gmail.com>

On Thursday, July 5, 2012 at 10:49 PM, Terry Reedy wrote:
> On 7/5/2012 6:48 PM, Donald Stufft wrote:
> > On Thursday, July 5, 2012 at 6:38 PM, Terry Reedy wrote:
> > > Last I knew, uploading a file required licensing it to the PSF. On the
> > > other hand, I can find no mention of that on
> > > http://pypi.python.org/pypi?%3Aaction=submit_form
> > > http://wiki.python.org/moin/CheeseShopTutorial
> > > nor a link to the license anywhere. So I don't know what the current
> > > situation is.
> > > 
> > 
> > http://www.python.org/about/legal/
> > 
> > IANAL but I think it pretty much it just says the things you upload to
> > the site, the site is allowed to let others download it and you don't
> > get to charge the PSF for it.
> > 
> 
> 
> That is pretty much what the first version says. The current version is 
> much more expansive and seems to deny any license restrictions.
> 
> "The Python Software Foundation ... has no obligation of any kind with 
> respect to such third party content."
> 
> not even to respect its license?
> 
> ... "The PSF is free to use or disseminate such content on an 
> unrestricted basis for any purpose,"
> 
> The purpose of any license to to restrict use or dissemination.
> 
> "and third party content providers grant the PSF and all other users of 
> the web site an irrevocable, worldwide, royalty-free, nonexclusive 
> license to reproduce, distribute, transmit, display, perform, and 
> publish such content, including in digital form."
> 
> That says to me that the PSF *and* its users are pretty much free of any 
> license restrictions on uploaded software, which negates the point of 
> having a license. For instance, without a declaration otherwise from the 
> FSF, I would not think it legal to upload a derivative of a GPL-licenced 
> work
> 
> ---
> Terry Jan Reedy
> 
> 

Because IANAL, and I'm going to guess most folks here are not either, I pinged VanL 
and forwarded him the conversationto ask him what the /about/terms/ means in terms
of software being uploaded to PyPI.

His response:

===============================================================
 
The issue is that we have several different use cases here - and one important one is
making mirrors of all the content in the repository for various uses. To keep everything
on the up-and-up, we make sure that we allow other people the right to redistribute the
content of PyPI in whatever way they see fit.

Terry brings up the issue of the GPL. That is a common question, but one that
misapprehends the nature of the rights granted under the GPL as well as to the PSF
for PyPI.

The GPL (in a nutshell) requires that you do two things:
1) Distribute the source along with the binary image.
2) License any derivative works under the GPL.

Note that the GPL explicitly says that there are no restrictions on use of the software.

We assume (without checking) that anyone uploading a GPL-licensed package will comply,
i.e., they will upload the source and the binaries together.

The PSF license allows anyone to use and redistribute what is uploaded. Therefore, if
someone mirrors PyPI, they will automatically be in compliance with obligation #1 of the
GPL as described above.

The issue is that people assume that the PSF or a third party can change the package
and only distribute the binaries. This is incorrect! The PSF license does not grant anybody
the right to create derivative works of the software - i.e., it doesn't grant people the right to
change what was uploaded and re-upload - unless they do so according to the terms of the
only license that grants them rights to create such a derivative work (i.e., the GPL). Therefore,
the only way to legally change what the original author provided is to comply with obligation
#2 of the GPL above.

Finally, I note that PyPI supports hosting python modules in many different locations. If
someone is unhappy with the above, they can host many other places and still have their
modules indexed on PyPI. Pip/easy_install will find the location of the binaries just fine.


===============================================================

So it sounds to me like that only license that the PSF /about/terms/ would clash with is one
that limits the right to distribute without modification the files uploaded to PyPI.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120706/3f842a36/attachment.html>

From kencochrane at gmail.com  Sat Jul  7 21:56:29 2012
From: kencochrane at gmail.com (ken cochrane)
Date: Sat, 7 Jul 2012 15:56:29 -0400
Subject: [Catalog-sig] Some pypi mirrors not up to date?
In-Reply-To: <CACBKDngEq8y5yFFPYo-Ymwd8f655tUviMXkw=JdheJhrfkW1Lg@mail.gmail.com>
References: <ipml5d$l4k$1@dough.gmane.org>
	<9B68E47E-1639-445D-AF5F-3791BB5CCF7D@leidel.info>
	<4DBF0FB4.5060900@v.loewis.de>
	<86BBA972-78A3-4D3C-A63E-FC327EF80E76@leidel.info>
	<CACBKDngEq8y5yFFPYo-Ymwd8f655tUviMXkw=JdheJhrfkW1Lg@mail.gmail.com>
Message-ID: <CACBKDnjVKgmMd4VCc8VhcUB2LuVppU0Yp8LMSVophc40NJTVpA@mail.gmail.com>

To keep everyone in the loop. I finally had time to sit down and work on
this, and I created a patch and submitted a pull request to Martin for his
review.

https://bitbucket.org/loewis/pypi-appengine/pull-request/1/added-some-logging-to-help-track-mirror

If anyone else has time to review the change as well, before I push it out,
that would be great. I'm still new to this code base, and I don't want to
break anything.

Ken


On Mon, Jul 2, 2012 at 8:37 AM, ken cochrane <kencochrane at gmail.com> wrote:

> Jannis,
>
> I started looking at this the other day, and I haven't had a chance to fix
> it because the Amazon datacenter outage took all of my time the past few
> days.
>
> Here is what I found out.
>
> b.pypi.python.org lives on GAE and it's currently stuck, and looking at
> the logs I figured out what went wrong, but I'm not sure how to fix it. [3]
> See log snippet at the end of the email.
>
> Basically there is a python package called '__past__'  (see [0] link
> below) that is causing the sync process to break because we are trying to
> use the project name as the key_name for the Product model [1], and GAE
> model key_name's can't contain underscores [2].
>
> I'm not sure how to fix the issue without possibly breaking other things.
> My first thought was to remove the underscores, but that might break
> something else, or conflict with another project with a similar name. I
> wrote to Martin who gave me the following advice.
>
> From "Martin v. L?wis":
>
> Renaming/escaping sounds good. I'd check if there is any string that
>> can be used in a GAE key name, but not be used in a PyPI package name.
>> If not, standard escaping needs to be applied: a prefix of "dunder"
>> is added to any package whose name starts and ends with __, as well
>> as to any package whose name starts with "dunder".
>>
>
>
>> When looking at all child nodes, remove "dunder" from any name;
>> when doing lookups by name, escape as above.
>>
>
>
>> If you do find a character/string that can be in a key name but
>> not in a package name, just escape the string with that name -
>> no need to worry about escaping the escape character. However it
>> may be that the only possible choice is "/" (which I know cannot
>> appear in a package name).
>
>
> I looked through most of the pypi code and I think the only character you
> can't have is "/", all other characters look like they work.
>
> So, I know what is causing it, we just need to fix the issue, test it, and
> roll out the fix. I was planning on doing it this past weekend but thanks
> to AWS, I didn't have any time to work on it. If anyone has any free time,
> feel free to take over / help. Just let others know so there isn't any
> duplicate effort.
>
> Let me know if you have any questions.
>
> Ken Cochrane
>
> Footnotes:
>
> [0] http://pypi.python.org/pypi/__past__/0.0.1.dev
>
> [1]
> https://bitbucket.org/loewis/pypi-appengine/src/fa6596a427e1/fetch.py#cl-62
>
>
> [2] Information about model key_names
>
> https://developers.google.com/appengine/docs/python/datastore/modelclass#Model
>  key_name
>
> The key name for the entity. The name becomes part of the primary key. If
> None, a system-generated numeric ID is used for the key.
>
> The value for key_name must not be of the form __*__.
> [1]
> Log snippet.
>
>    1. 2012-06-28 06:45:18.222
>
>    step package '__past__'
>
>    2. E2012-06-28 06:45:18.778
>
>    illegal name in key path element: __past__
>    Traceback (most recent call last):
>      File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
>        handler.get(*groups)
>      File "/base/data/home/apps/pypi/3.358089379617981219/handlers.py", line 171, in get
>        self.response.out.write(fetch.cron())
>      File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py", line 293, in cron
>        return step()
>      File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py", line 259, in step
>        actions[action](m, todo, param)
>      File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py", line 91, in package
>        data = simple_page(m, name)
>      File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py", line 70, in simple_page
>        obj.put()
>      File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1074, in put
>        return datastore.Put(self._entity, **kwargs)
>      File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 579, in Put
>        return PutAsync(entities, **kwargs).get_result()
>      File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
>        return self.__get_result_hook(self)
>      File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1579, in __put_hook
>        self.check_rpc_success(rpc)
>      File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1216, in check_rpc_success
>        raise _ToDatastoreError(err)
>    BadRequestError: illegal name in key path element: __past__
>
>
>
>
> On Mon, Jul 2, 2012 at 8:11 AM, Jannis Leidel <jannis at leidel.info> wrote:
>
>> On 02.05.2011, at 22:10, Martin v. L?wis <martin at v.loewis.de> wrote:
>>
>> > Am 02.05.2011 19:24, schrieb Jannis Leidel:
>> >>
>> >> On 02.05.2011, at 18:12, Maurits van Rees wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I noticed that some distributions are not on all mirrors.  For example
>> >>> http://a.pypi.python.org/simple/plone.app.referenceablebehavior/
>> >>> has 0.1 and 0.2 (last one released 30 April)
>> >>> but 0.2 is missing from
>> >>> http://b.pypi.python.org/simple/plone.app.referenceablebehavior/
>> >>>
>> >>> Same for c and d.  Ah, no: those two have it now.  I know for sure
>> that at least d did not have it five minutes ago.  And this version has
>> been released two days ago, so it should have been slightly faster. :-)
>> >>
>> >> Hm, d doesn't seem to have the file on disk even thought it's on the
>> simple page, see
>> http://d.pypi.python.org/packages/source/p/plone.app.referenceablebehavior/
>> >>
>> >> Martin: Anything I can do to make sure this doesn't happen again?
>> >
>> > As the starting point, we should figure out why it happened in
>> > the first place - it shouldn't have, of course. Most likely,
>> > it's a bug :-)
>>
>> Looks like http://b.pypi.python.org is out of date again:
>> http://www.pypi-mirrors.org
>>
>> Can we do something about that?
>>
>> Jannis
>>
>> _______________________________________________
>> Catalog-SIG mailing list
>> Catalog-SIG at python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120707/a3d7d8bf/attachment.html>

From martin at v.loewis.de  Sun Jul  8 19:30:52 2012
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 08 Jul 2012 19:30:52 +0200
Subject: [Catalog-sig] Feature idea for PyPI third-party projects
In-Reply-To: <09887482C1B04D82894EFD3FC600530B@gmail.com>
References: <4FF547E0.4050000@ziade.org>
	<B87A41C8870F449B8AE00C52055B92B3@gmail.com>
	<4FF612E5.2000909@ziade.org>
	<20120706004738.Horde.GqqLL8L8999P9hmKn4iWvLA@webmail.df.eu>
	<09887482C1B04D82894EFD3FC600530B@gmail.com>
Message-ID: <4FF9C3CC.5090701@v.loewis.de>

On 06.07.2012 00:50, Donald Stufft wrote:
> On Thursday, July 5, 2012 at 6:47 PM, martin at v.loewis.de wrote:
>>> Markus mentioned he'd rather have a rss feed
>>>
>>> So maybe being able to do a project selection, then create a custom
>>> feed and/or a mail per publication
>>
>> I'd like to point out that PyPI support pubsubhubbub. So any of these
>> services can be built on to of that.
> I had forgotten about pubsubhubbub, I need to look at that again to
> try and lessen the lag time in replication more. Do you happen to have
> a link for it? 

The hub is

http://pubsubhubbub.appspot.com/publish

and the topic is

http://pypi.python.org/pypi?:action=lasthour

See also

http://www.zope.de/tagung/Dresden_2010/shop-apis_mvl.pdf

Regards,
Martin

From donald.stufft at gmail.com  Sun Jul  8 22:50:29 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Sun, 8 Jul 2012 16:50:29 -0400
Subject: [Catalog-sig] Feature idea for PyPI third-party projects
In-Reply-To: <4FF9C3CC.5090701@v.loewis.de>
References: <4FF547E0.4050000@ziade.org>
	<B87A41C8870F449B8AE00C52055B92B3@gmail.com>
	<4FF612E5.2000909@ziade.org>
	<20120706004738.Horde.GqqLL8L8999P9hmKn4iWvLA@webmail.df.eu>
	<09887482C1B04D82894EFD3FC600530B@gmail.com>
	<4FF9C3CC.5090701@v.loewis.de>
Message-ID: <F8BDAD377D8D4F05BBF26F62152F49FC@gmail.com>

On Sunday, July 8, 2012 at 1:30 PM, "Martin v. L?wis" wrote:
> The hub is
>  
> http://pubsubhubbub.appspot.com/publish
>  
> and the topic is
>  
> http://pypi.python.org/pypi?:action=lasthour
>  
> See also
>  
> http://www.zope.de/tagung/Dresden_2010/shop-apis_mvl.pdf
>  
> Regards,
> Martin
>  
>  

Thanks!  

Now to plug this into google translate ;)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120708/d5fcf0e6/attachment.html>

From aclark at aclark.net  Fri Jul 20 04:39:28 2012
From: aclark at aclark.net (Alex Clark)
Date: Thu, 19 Jul 2012 22:39:28 -0400
Subject: [Catalog-sig] pythonpackages.com beta security
Message-ID: <juagd1$icn$1@dough.gmane.org>

Hi,


Earlier in the year I announced the pythonpackages.com alpha[1] and 
there was some helpful discussion in that thread about security. We are 
now in beta and since then, I've

- totally abandoned the idea of using pypissh
- investigated using PyPI oauth[3], but gave up
- settled on saving users credentials in an encrypted session cookie[4]

While not ideal, I'm fairly happy with the fact that it works and is 
"secure enough". However I'd very much appreciate some additional eyes 
on the implementation. The (very simple) pyramid code is:


     if 'submit' in request.POST:
         username = request.POST['username']
         password = request.POST['password']
         cookieval = {'username': username, 'password': password}
         request.session[config.COOKIE_PYPI] = cookieval


And the beta is available to anyone who signs up here: 
http://pythonpackages.com/signup, then signs in with their GitHub 
account. You can then go here:

- https://pythonpackages.com/manage/account/pypi

and fill in your PyPI credentials (or bogus credentials for testing) 
then verify you cannot easily extract the account info from the 
beaker.session.id cookie


Thanks for any feedback,


Alex


[1] http://mail.python.org/pipermail/catalog-sig/2012-January/004152.html

[2] http://pypi.python.org/pypi/pypissh/1.4

[3] https://bitbucket.org/loewis/pypi/changeset/b034fda5bef9

[4] http://beaker.readthedocs.org/en/latest/sessions.html#encryption



-- 
Alex Clark ? http://pythonpackages.com/ONE_CLICK_RELEASE


From richard at python.org  Fri Jul 20 06:20:43 2012
From: richard at python.org (Richard Jones)
Date: Fri, 20 Jul 2012 14:20:43 +1000
Subject: [Catalog-sig] pythonpackages.com beta security
In-Reply-To: <juagd1$icn$1@dough.gmane.org>
References: <juagd1$icn$1@dough.gmane.org>
Message-ID: <CAHrZfZAnE8JsEXxZ3W+PoEbeWZFBrT22SehfYmSy0HnUmUogFw@mail.gmail.com>

We implemented OAuth for you and crate.io. Why did you give up?


    Richard

On 20 July 2012 12:39, Alex Clark <aclark at aclark.net> wrote:
> Hi,
>
>
> Earlier in the year I announced the pythonpackages.com alpha[1] and there
> was some helpful discussion in that thread about security. We are now in
> beta and since then, I've
>
> - totally abandoned the idea of using pypissh
> - investigated using PyPI oauth[3], but gave up
> - settled on saving users credentials in an encrypted session cookie[4]
>
> While not ideal, I'm fairly happy with the fact that it works and is "secure
> enough". However I'd very much appreciate some additional eyes on the
> implementation. The (very simple) pyramid code is:
>
>
>     if 'submit' in request.POST:
>         username = request.POST['username']
>         password = request.POST['password']
>         cookieval = {'username': username, 'password': password}
>         request.session[config.COOKIE_PYPI] = cookieval
>
>
> And the beta is available to anyone who signs up here:
> http://pythonpackages.com/signup, then signs in with their GitHub account.
> You can then go here:
>
> - https://pythonpackages.com/manage/account/pypi
>
> and fill in your PyPI credentials (or bogus credentials for testing) then
> verify you cannot easily extract the account info from the beaker.session.id
> cookie
>
>
> Thanks for any feedback,
>
>
> Alex
>
>
> [1] http://mail.python.org/pipermail/catalog-sig/2012-January/004152.html
>
> [2] http://pypi.python.org/pypi/pypissh/1.4
>
> [3] https://bitbucket.org/loewis/pypi/changeset/b034fda5bef9
>
> [4] http://beaker.readthedocs.org/en/latest/sessions.html#encryption
>
>
>
> --
> Alex Clark ? http://pythonpackages.com/ONE_CLICK_RELEASE
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>

From aclark at aclark.net  Fri Jul 20 06:37:42 2012
From: aclark at aclark.net (Alex Clark)
Date: Fri, 20 Jul 2012 00:37:42 -0400
Subject: [Catalog-sig] pythonpackages.com beta security
In-Reply-To: <CAHrZfZAnE8JsEXxZ3W+PoEbeWZFBrT22SehfYmSy0HnUmUogFw@mail.gmail.com>
References: <juagd1$icn$1@dough.gmane.org>
	<CAHrZfZAnE8JsEXxZ3W+PoEbeWZFBrT22SehfYmSy0HnUmUogFw@mail.gmail.com>
Message-ID: <juanam$qv2$1@dough.gmane.org>

On 7/20/12 12:20 AM, Richard Jones wrote:
> We implemented OAuth for you and crate.io. Why did you give up?


Perhaps "give up" is not the right description. I postponed until I can 
figure it out, and went with an encrypted session cookie in the 
meantime, hoping it would be safe enough and that users would go for it.


As for oauth, at least in my case, I'm looking for something that can be 
implemented very simply e.g.:


- http://developer.github.com/v3/oauth/


(my implementation is done totally with requests)


IIUC, there are no docs for the PyPI implementation of oauth and it's 
oauth1? (vs oauth2)? If it currently works similar to GitHub's 
implementation, then I should take another look. If it doesn't, then we 
should talk about the details.



Alex




>
>
>      Richard
>
> On 20 July 2012 12:39, Alex Clark <aclark at aclark.net> wrote:
>> Hi,
>>
>>
>> Earlier in the year I announced the pythonpackages.com alpha[1] and there
>> was some helpful discussion in that thread about security. We are now in
>> beta and since then, I've
>>
>> - totally abandoned the idea of using pypissh
>> - investigated using PyPI oauth[3], but gave up
>> - settled on saving users credentials in an encrypted session cookie[4]
>>
>> While not ideal, I'm fairly happy with the fact that it works and is "secure
>> enough". However I'd very much appreciate some additional eyes on the
>> implementation. The (very simple) pyramid code is:
>>
>>
>>      if 'submit' in request.POST:
>>          username = request.POST['username']
>>          password = request.POST['password']
>>          cookieval = {'username': username, 'password': password}
>>          request.session[config.COOKIE_PYPI] = cookieval
>>
>>
>> And the beta is available to anyone who signs up here:
>> http://pythonpackages.com/signup, then signs in with their GitHub account.
>> You can then go here:
>>
>> - https://pythonpackages.com/manage/account/pypi
>>
>> and fill in your PyPI credentials (or bogus credentials for testing) then
>> verify you cannot easily extract the account info from the beaker.session.id
>> cookie
>>
>>
>> Thanks for any feedback,
>>
>>
>> Alex
>>
>>
>> [1] http://mail.python.org/pipermail/catalog-sig/2012-January/004152.html
>>
>> [2] http://pypi.python.org/pypi/pypissh/1.4
>>
>> [3] https://bitbucket.org/loewis/pypi/changeset/b034fda5bef9
>>
>> [4] http://beaker.readthedocs.org/en/latest/sessions.html#encryption
>>
>>
>>
>> --
>> Alex Clark ? http://pythonpackages.com/ONE_CLICK_RELEASE
>>
>> _______________________________________________
>> Catalog-SIG mailing list
>> Catalog-SIG at python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig
>>


-- 
Alex Clark ? http://pythonpackages.com/ONE_CLICK_RELEASE


From richard at python.org  Fri Jul 20 07:07:09 2012
From: richard at python.org (Richard Jones)
Date: Fri, 20 Jul 2012 15:07:09 +1000
Subject: [Catalog-sig] pythonpackages.com beta security
In-Reply-To: <juanam$qv2$1@dough.gmane.org>
References: <juagd1$icn$1@dough.gmane.org>
	<CAHrZfZAnE8JsEXxZ3W+PoEbeWZFBrT22SehfYmSy0HnUmUogFw@mail.gmail.com>
	<juanam$qv2$1@dough.gmane.org>
Message-ID: <CAHrZfZCUiJupXt6kaHojZDYp8voVQfK+xzb7dWVxfsbR6ir0SA@mail.gmail.com>

On 20 July 2012 14:37, Alex Clark <aclark at aclark.net> wrote:
> As for oauth, at least in my case, I'm looking for something that can be
> implemented very simply e.g.:
>
>
> - http://developer.github.com/v3/oauth/

That's OAuth2, which is quite unlike the OAuth1(a) that we
implemented. You cannot do OAuth1 with just requests, as far as I'm
aware. There's no documentation for PyPI OAuth as we're still waiting
for it to be used by someone to prove its usefulness.

There is, however, sample code in the pypi repository in the
oauth_test.py file that exercises the API. It uses the common oauth
library* with a convenience wrapper. This code is all you would need
to use the OAuth PyPI facility. I would be more than happy to help you
make use of it.


    Richard

* http://pypi.python.org/pypi/oauth

From donald.stufft at gmail.com  Fri Jul 20 07:43:38 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Fri, 20 Jul 2012 01:43:38 -0400
Subject: [Catalog-sig] pythonpackages.com beta security
In-Reply-To: <CAHrZfZCUiJupXt6kaHojZDYp8voVQfK+xzb7dWVxfsbR6ir0SA@mail.gmail.com>
References: <juagd1$icn$1@dough.gmane.org>
	<CAHrZfZAnE8JsEXxZ3W+PoEbeWZFBrT22SehfYmSy0HnUmUogFw@mail.gmail.com>
	<juanam$qv2$1@dough.gmane.org>
	<CAHrZfZCUiJupXt6kaHojZDYp8voVQfK+xzb7dWVxfsbR6ir0SA@mail.gmail.com>
Message-ID: <AD0C23AE51FA47ACB6BFABFCCCAB8A1B@gmail.com>

On Friday, July 20, 2012 at 1:07 AM, Richard Jones wrote:
> > 
> 
> That's OAuth2, which is quite unlike the OAuth1(a) that we
> implemented. You cannot do OAuth1 with just requests, as far as I'm
> aware. There's no documentation for PyPI OAuth as we're still waiting
> for it to be used by someone to prove its usefulness.
> 
> 

I havn't had a chance yet (doing a major refactor first), but there's experimental
OAuth 1a support in the most recent versions of requests. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120720/f3500ebd/attachment.html>

From richard at python.org  Fri Jul 20 08:38:31 2012
From: richard at python.org (Richard Jones)
Date: Fri, 20 Jul 2012 16:38:31 +1000
Subject: [Catalog-sig] pythonpackages.com beta security
In-Reply-To: <AD0C23AE51FA47ACB6BFABFCCCAB8A1B@gmail.com>
References: <juagd1$icn$1@dough.gmane.org>
	<CAHrZfZAnE8JsEXxZ3W+PoEbeWZFBrT22SehfYmSy0HnUmUogFw@mail.gmail.com>
	<juanam$qv2$1@dough.gmane.org>
	<CAHrZfZCUiJupXt6kaHojZDYp8voVQfK+xzb7dWVxfsbR6ir0SA@mail.gmail.com>
	<AD0C23AE51FA47ACB6BFABFCCCAB8A1B@gmail.com>
Message-ID: <CAHrZfZBH=zmh-Cf3YaouA-5+a_8DUqrQzBNHSprV1c1ZdCo_cQ@mail.gmail.com>

On 20 July 2012 15:43, Donald Stufft <donald.stufft at gmail.com> wrote:
> On Friday, July 20, 2012 at 1:07 AM, Richard Jones wrote:
>
> That's OAuth2, which is quite unlike the OAuth1(a) that we
> implemented. You cannot do OAuth1 with just requests, as far as I'm
> aware. There's no documentation for PyPI OAuth as we're still waiting
> for it to be used by someone to prove its usefulness.
>
> I havn't had a chance yet (doing a major refactor first), but there's
> experimental
> OAuth 1a support in the most recent versions of requests.

Oh, nice! I'll see if I can find some time over the weekend to write
up how to use that against the PyPI implementation.


    Richard

From aclark at aclark.net  Fri Jul 20 18:37:30 2012
From: aclark at aclark.net (Alex Clark)
Date: Fri, 20 Jul 2012 12:37:30 -0400
Subject: [Catalog-sig] pythonpackages.com beta security
In-Reply-To: <CAHrZfZBH=zmh-Cf3YaouA-5+a_8DUqrQzBNHSprV1c1ZdCo_cQ@mail.gmail.com>
References: <juagd1$icn$1@dough.gmane.org>
	<CAHrZfZAnE8JsEXxZ3W+PoEbeWZFBrT22SehfYmSy0HnUmUogFw@mail.gmail.com>
	<juanam$qv2$1@dough.gmane.org>
	<CAHrZfZCUiJupXt6kaHojZDYp8voVQfK+xzb7dWVxfsbR6ir0SA@mail.gmail.com>
	<AD0C23AE51FA47ACB6BFABFCCCAB8A1B@gmail.com>
	<CAHrZfZBH=zmh-Cf3YaouA-5+a_8DUqrQzBNHSprV1c1ZdCo_cQ@mail.gmail.com>
Message-ID: <juc1gb$9od$1@dough.gmane.org>

Hi,

On 7/20/12 2:38 AM, Richard Jones wrote:
> On 20 July 2012 15:43, Donald Stufft <donald.stufft at gmail.com> wrote:
>> On Friday, July 20, 2012 at 1:07 AM, Richard Jones wrote:
>>
>> That's OAuth2, which is quite unlike the OAuth1(a) that we
>> implemented. You cannot do OAuth1 with just requests, as far as I'm
>> aware. There's no documentation for PyPI OAuth as we're still waiting
>> for it to be used by someone to prove its usefulness.
>>
>> I havn't had a chance yet (doing a major refactor first), but there's
>> experimental
>> OAuth 1a support in the most recent versions of requests.
>
> Oh, nice! I'll see if I can find some time over the weekend to write
> up how to use that against the PyPI implementation.


Nice indeed! I'll take a look, too. Assuming I can get oauth1 going, 
will that allow me to make releases on behalf of users? I'm not sure if 
this is an oauth1 or 2 thing, but on GitHub you can choose which "scope" 
you want your application to ask the user to grant to it:


- http://developer.github.com/v3/oauth/#scopes


So at the very least, I'd like my application to enable users to do the 
equivalent of distutils' register and upload commands. The workflow 
looks something like this:


- Create package via PasteScript-powered web form: 
https://pythonpackages.com/manage/package/new

- Clone, develop code locally, and push

- Test the package release on pythonpackages.com via web form 
submissions that execute the following:

   $ python setup.py install
   $ python setup.py sdist upload -r http://index.pythonpackages.com

- Manually test the release locally via:

   $ pip install PACKAGE -i http://index.pythonpackages.com

- Release the package to PyPI via

   $ python setup.py register sdist upload




Alex





>
>
>      Richard
>


-- 
Alex Clark ? http://pythonpackages.com/ONE_CLICK_RELEASE


From techtonik at gmail.com  Sun Jul 22 14:39:50 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Sun, 22 Jul 2012 15:39:50 +0300
Subject: [Catalog-sig] PyPI pull request
In-Reply-To: <CAPkN8x+pMdCK_u-HknR2=wsK+DwKg62798O9DR0JBFqpBQakXA@mail.gmail.com>
References: <CAPkN8xJ+2C9Thp8J7BBg06kRpF+XF2CM89OBsmP4XsBi28kicg@mail.gmail.com>
	<CAPkN8x+pMdCK_u-HknR2=wsK+DwKg62798O9DR0JBFqpBQakXA@mail.gmail.com>
Message-ID: <CAPkN8xJv4_JdeNgk7d4ZFN+Jrx4d07vdJE6j0DbgEyYZ34Xj1Q@mail.gmail.com>

On Mon, May 28, 2012 at 2:29 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> On Wed, May 9, 2012 at 1:51 PM, anatoly techtonik <techtonik at gmail.com> wrote:
>>
>> Anybody to merge my changes from
>> https://bitbucket.org/techtonik/pypi-techtonik ?
>
> Anybody?

That suxx, but I still can not get any reply from Martin after several
reminders, so I'm rejecting my pull request. At least next time then
somebody says that I only complain, I have the argument that it is
damn HARD to push something through you guys.

https://bitbucket.org/loewis/pypi/pull-request/1/fix-imports-add-logging-to-console-in
--
anatoly t.

From tarek at ziade.org  Wed Jul 25 12:25:06 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 25 Jul 2012 12:25:06 +0200
Subject: [Catalog-sig] Super heavy index page
Message-ID: <500FC982.6030902@ziade.org>

Hey

I would like to suggest redirecting http://pypi.python.org/pypi/ to 
http://pypi.python.org/pypi

http://pypi.python.org/pypi/ is *very* heavy - as it pulls all pypi 
packages on the same page. It freezes my browser completely for a few 
seconds :D

Cheers
Tarek

From hanno at hannosch.eu  Wed Jul 25 13:15:53 2012
From: hanno at hannosch.eu (Hanno Schlichting)
Date: Wed, 25 Jul 2012 13:15:53 +0200
Subject: [Catalog-sig] Super heavy index page
In-Reply-To: <500FC982.6030902@ziade.org>
References: <500FC982.6030902@ziade.org>
Message-ID: <CAJ5sox44+wnXhUq5MpQkAdo9EFkKzPbx2Vd90NVcq86Nvp+=uQ@mail.gmail.com>

On Wed, Jul 25, 2012 at 12:25 PM, Tarek Ziad? <tarek at ziade.org> wrote:
> I would like to suggest redirecting http://pypi.python.org/pypi/ to
> http://pypi.python.org/pypi
>
> http://pypi.python.org/pypi/ is *very* heavy - as it pulls all pypi packages
> on the same page. It freezes my browser completely for a few seconds :D

IIRC we did this for the benefit of setuptools, which used to use the
trailing slash version as its index page.

It switched to the /simple view more than four years ago and this
change is already in setuptools 0.6c9.

I think it's therefor safe to remove this special case.

Hanno

From martin at v.loewis.de  Wed Jul 25 14:48:07 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 25 Jul 2012 14:48:07 +0200
Subject: [Catalog-sig] Super heavy index page
In-Reply-To: <500FC982.6030902@ziade.org>
References: <500FC982.6030902@ziade.org>
Message-ID: <20120725144807.Horde.D3wsPKGZi1VQD_sHfD1wkqA@webmail.df.eu>

> I would like to suggest redirecting http://pypi.python.org/pypi/ to  
> http://pypi.python.org/pypi

This would break tools who use that URL to fetch the complete list of
packages. Old setuptools versions did; not sure whether they are still
in use, or whether any other tools fetch the package list from there.

Regards,
Martin



From tarek at ziade.org  Wed Jul 25 15:11:23 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 25 Jul 2012 15:11:23 +0200
Subject: [Catalog-sig] Super heavy index page
In-Reply-To: <20120725144807.Horde.D3wsPKGZi1VQD_sHfD1wkqA@webmail.df.eu>
References: <500FC982.6030902@ziade.org>
	<20120725144807.Horde.D3wsPKGZi1VQD_sHfD1wkqA@webmail.df.eu>
Message-ID: <500FF07B.6000900@ziade.org>

On 7/25/12 2:48 PM, martin at v.loewis.de wrote:
>> I would like to suggest redirecting http://pypi.python.org/pypi/ to 
>> http://pypi.python.org/pypi
>
> This would break tools who use that URL to fetch the complete list of
> packages. Old setuptools versions did; not sure whether they are still
> in use, or whether any other tools fetch the package list from there.

Maybe the User-Agent header can be used to make that decision ?

All urllib2-based code have the same kind of user agent

>
> Regards,
> Martin
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig



From tjreedy at udel.edu  Wed Jul 25 22:20:32 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 25 Jul 2012 16:20:32 -0400
Subject: [Catalog-sig] Super heavy index page
In-Reply-To: <20120725144807.Horde.D3wsPKGZi1VQD_sHfD1wkqA@webmail.df.eu>
References: <500FC982.6030902@ziade.org>
	<20120725144807.Horde.D3wsPKGZi1VQD_sHfD1wkqA@webmail.df.eu>
Message-ID: <jupkf5$mjm$1@dough.gmane.org>

On 7/25/2012 8:48 AM, martin at v.loewis.de wrote:
>> I would like to suggest redirecting http://pypi.python.org/pypi/ to
>> http://pypi.python.org/pypi

For people, the latter has a [List packages] button that maps to
http://pypi.python.org/pypi?action=index

> This would break tools who use that URL to fetch the complete list of
> packages. Old setuptools versions did; not sure whether they are still
> in use, or whether any other tools fetch the package list from there.

Too bad the tools were not taught to use the synonym above (assuming it 
would work for them). Would it be possible for the server to immediately 
return the boilerplate sidebar and header before returning the sorted 
list a few seconds later. For people, it would be more reassuring to see 
a partial response immediately.

It seems to me that tools would be better off with just the list itself 
without the surrounding stuff that they have to ignore (filter out). 
Also, tools could be sent an unsorted list that they could sort locally 
if they needed it. I suspect most do not, or only need sorting after 
filtering.

-- 
Terry Jan Reedy




From martin at v.loewis.de  Wed Jul 25 22:46:39 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 25 Jul 2012 22:46:39 +0200
Subject: [Catalog-sig] Super heavy index page
In-Reply-To: <jupkf5$mjm$1@dough.gmane.org>
References: <500FC982.6030902@ziade.org>
	<20120725144807.Horde.D3wsPKGZi1VQD_sHfD1wkqA@webmail.df.eu>
	<jupkf5$mjm$1@dough.gmane.org>
Message-ID: <50105B2F.80607@v.loewis.de>

> Too bad the tools were not taught to use the synonym above (assuming it
> would work for them). Would it be possible for the server to immediately
> return the boilerplate sidebar and header before returning the sorted
> list a few seconds later. For people, it would be more reassuring to see
> a partial response immediately.
> 
> It seems to me that tools would be better off with just the list itself
> without the surrounding stuff that they have to ignore (filter out).
> Also, tools could be sent an unsorted list that they could sort locally
> if they needed it. I suspect most do not, or only need sorting after
> filtering.

Tools might have switched to http://pypi.python.org/simple/ several
years ago, then they would have gotten all what you are proposing
already.

I don't see how your reasoning helps in the current state, assuming
some tools still do access /pypi/.

I just checked the logs: yesterday, there were 233 accesses to the URL
from user agents with "urllib" in their names, mostly

"Python-urllib/2.5 setuptools/0.6c5"
"Python-urllib/2.7"
"Python-urllib/2.7 distribute/0.6.27"

There are also some interesting referrers:

http://www.python.org/download/ (just fixed)
http://wiki.zope.org/zope3/QuickInstallGuide

Besides spiders and regular browsers, the only non-urllib access came from

"AppEngine-Google; (+http://code.google.com/appengine; appid: feediscovery)"


Regards,
Martin


From tjreedy at udel.edu  Wed Jul 25 22:59:47 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 25 Jul 2012 16:59:47 -0400
Subject: [Catalog-sig] Super heavy index page
In-Reply-To: <50105B2F.80607@v.loewis.de>
References: <500FC982.6030902@ziade.org>
	<20120725144807.Horde.D3wsPKGZi1VQD_sHfD1wkqA@webmail.df.eu>
	<jupkf5$mjm$1@dough.gmane.org> <50105B2F.80607@v.loewis.de>
Message-ID: <jupmoo$8ui$1@dough.gmane.org>

On 7/25/2012 4:46 PM, "Martin v. L?wis" wrote:

I just checked, and pypi.python.org, which is what users should be typing,
does the right thing and goes to /pypi.

> http://www.python.org/download/ (just fixed)

With that fixed (I just tried it), I agree that there is nothing more to 
do here. I think we can assume that some people who type or bookmark 
/pypi/ actually want the full list.

> http://wiki.zope.org/zope3/QuickInstallGuide

That is zope responsibility. Anatoly, if you think they have the wrong 
link, take it up with them. I think we can assume that some people who 
type or bookmark /pypi/ actually want the full list.

-- 
Terry Jan Reedy





From fred at fdrake.net  Wed Jul 25 23:18:57 2012
From: fred at fdrake.net (Fred Drake)
Date: Wed, 25 Jul 2012 17:18:57 -0400
Subject: [Catalog-sig] Super heavy index page
In-Reply-To: <jupmoo$8ui$1@dough.gmane.org>
References: <500FC982.6030902@ziade.org>
	<20120725144807.Horde.D3wsPKGZi1VQD_sHfD1wkqA@webmail.df.eu>
	<jupkf5$mjm$1@dough.gmane.org> <50105B2F.80607@v.loewis.de>
	<jupmoo$8ui$1@dough.gmane.org>
Message-ID: <CAFT4OTGRC8CZPaedxZP2EARTDSWVzJy1p=yBtaz=rZbNyYeSHQ@mail.gmail.com>

On Wed, Jul 25, 2012 at 4:59 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> http://wiki.zope.org/zope3/QuickInstallGuide
>
>
> That is zope responsibility. Anatoly, if you think they have the wrong link,
> take it up with them. I think we can assume that some people who type or
> bookmark /pypi/ actually want the full list.

That old wiki is in read-only mode, so it'll take some work to edit.

There's nothing there that cares about which version of the PyPI front
page is loaded; the short version would be fine.  Whether that's done by
a redirect or changing what's rendered doesn't matter as far as we're
concerned on this end.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From r1chardj0n3s at gmail.com  Fri Jul 27 03:17:12 2012
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Fri, 27 Jul 2012 11:17:12 +1000
Subject: [Catalog-sig] PyPI OAuth documentation
Message-ID: <CAHrZfZCAeYZo0a459-z-nAem+=m-DJTCXMsca0nuTkSUwS6-jQ@mail.gmail.com>

Hi all,

In order to help things along I've written up some documentation for
using the PyPI OAuth interface with requests and rauth:

http://wiki.python.org/moin/PyPiOauth

Note that you need to use my branch of requests for this to work
(https://github.com/r1chardj0n3s/requests) and rauth currently needs
to be modified to handle unverifiable SSL certificates.


   Richard

From r1chardj0n3s at gmail.com  Fri Jul 27 03:34:03 2012
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Fri, 27 Jul 2012 11:34:03 +1000
Subject: [Catalog-sig] PyPI OAuth documentation
In-Reply-To: <CAHrZfZCAeYZo0a459-z-nAem+=m-DJTCXMsca0nuTkSUwS6-jQ@mail.gmail.com>
References: <CAHrZfZCAeYZo0a459-z-nAem+=m-DJTCXMsca0nuTkSUwS6-jQ@mail.gmail.com>
Message-ID: <CAHrZfZCO+y0Upb6i77cWPSKLTvizxs8bq4hKT+aH-Oon8Kn2dg@mail.gmail.com>

On 27 July 2012 11:17, Richard Jones <r1chardj0n3s at gmail.com> wrote:
> Note that you need to use my branch of requests for this to work
> (https://github.com/r1chardj0n3s/requests) and rauth currently needs
> to be modified to handle unverifiable SSL certificates.

The author of rauth has shown me how to avoid the modification; the
code on the wiki page now works with the current un-modified rauth
release.


    Richard

From dholth at gmail.com  Fri Jul 27 03:45:26 2012
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 26 Jul 2012 21:45:26 -0400
Subject: [Catalog-sig] PyPI OAuth documentation
In-Reply-To: <CAHrZfZCO+y0Upb6i77cWPSKLTvizxs8bq4hKT+aH-Oon8Kn2dg@mail.gmail.com>
References: <CAHrZfZCAeYZo0a459-z-nAem+=m-DJTCXMsca0nuTkSUwS6-jQ@mail.gmail.com>
	<CAHrZfZCO+y0Upb6i77cWPSKLTvizxs8bq4hKT+aH-Oon8Kn2dg@mail.gmail.com>
Message-ID: <A8C6ABC4-8E61-4D36-AF8C-7BF6944140AB@gmail.com>

You can also pass the expected ssl certificate to requests, but I'm not sure of the exact syntax (see the advanced use section of their docs). This is the most secure option if you know which cert the server should present.

Daniel Holth

On Jul 26, 2012, at 9:34 PM, Richard Jones <r1chardj0n3s at gmail.com> wrote:

> On 27 July 2012 11:17, Richard Jones <r1chardj0n3s at gmail.com> wrote:
>> Note that you need to use my branch of requests for this to work
>> (https://github.com/r1chardj0n3s/requests) and rauth currently needs
>> to be modified to handle unverifiable SSL certificates.
> 
> The author of rauth has shown me how to avoid the modification; the
> code on the wiki page now works with the current un-modified rauth
> release.
> 
> 
>    Richard
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig

From r1chardj0n3s at gmail.com  Fri Jul 27 08:36:14 2012
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Fri, 27 Jul 2012 16:36:14 +1000
Subject: [Catalog-sig] PyPI OAuth documentation
In-Reply-To: <A8C6ABC4-8E61-4D36-AF8C-7BF6944140AB@gmail.com>
References: <CAHrZfZCAeYZo0a459-z-nAem+=m-DJTCXMsca0nuTkSUwS6-jQ@mail.gmail.com>
	<CAHrZfZCO+y0Upb6i77cWPSKLTvizxs8bq4hKT+aH-Oon8Kn2dg@mail.gmail.com>
	<A8C6ABC4-8E61-4D36-AF8C-7BF6944140AB@gmail.com>
Message-ID: <CAHrZfZD8bbtyaj=EYGGxMYq5nrdiE=f4LxXbWC1xe8Y80PjS5A@mail.gmail.com>

I'm not sure we can securely distribute the PEM to all the potential
users. A better solution is to have a Real SSL Certificate, but that
effort keeps failing :-(

On the other hand, Kenneth just released a new version of requests
which removes the need to use my branch :-)


    Richard

On 27 July 2012 11:45, Daniel Holth <dholth at gmail.com> wrote:
> You can also pass the expected ssl certificate to requests, but I'm not sure of the exact syntax (see the advanced use section of their docs). This is the most secure option if you know which cert the server should present.
>
> Daniel Holth
>
> On Jul 26, 2012, at 9:34 PM, Richard Jones <r1chardj0n3s at gmail.com> wrote:
>
>> On 27 July 2012 11:17, Richard Jones <r1chardj0n3s at gmail.com> wrote:
>>> Note that you need to use my branch of requests for this to work
>>> (https://github.com/r1chardj0n3s/requests) and rauth currently needs
>>> to be modified to handle unverifiable SSL certificates.
>>
>> The author of rauth has shown me how to avoid the modification; the
>> code on the wiki page now works with the current un-modified rauth
>> release.
>>
>>
>>    Richard
>> _______________________________________________
>> Catalog-SIG mailing list
>> Catalog-SIG at python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig

From dholth at gmail.com  Fri Jul 27 12:52:26 2012
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 27 Jul 2012 06:52:26 -0400
Subject: [Catalog-sig] PyPI OAuth documentation
In-Reply-To: <CAHrZfZD8bbtyaj=EYGGxMYq5nrdiE=f4LxXbWC1xe8Y80PjS5A@mail.gmail.com>
References: <CAHrZfZCAeYZo0a459-z-nAem+=m-DJTCXMsca0nuTkSUwS6-jQ@mail.gmail.com>
	<CAHrZfZCO+y0Upb6i77cWPSKLTvizxs8bq4hKT+aH-Oon8Kn2dg@mail.gmail.com>
	<A8C6ABC4-8E61-4D36-AF8C-7BF6944140AB@gmail.com>
	<CAHrZfZD8bbtyaj=EYGGxMYq5nrdiE=f4LxXbWC1xe8Y80PjS5A@mail.gmail.com>
Message-ID: <9307D049-3E7E-4EE3-9A1F-B6E6420E590E@gmail.com>

If you take a look at the "convergence" system and make sure different parts of the net see the same key, or just do what ssh does and assume the key won't change after the first time, those are good solutions. Of course it really should just have a ca signature too.

Daniel Holth

On Jul 27, 2012, at 2:36 AM, Richard Jones <r1chardj0n3s at gmail.com> wrote:

> I'm not sure we can securely distribute the PEM to all the potential
> users. A better solution is to have a Real SSL Certificate, but that
> effort keeps failing :-(
> 
> On the other hand, Kenneth just released a new version of requests
> which removes the need to use my branch :-)
> 
> 
>    Richard
> 
> On 27 July 2012 11:45, Daniel Holth <dholth at gmail.com> wrote:
>> You can also pass the expected ssl certificate to requests, but I'm not sure of the exact syntax (see the advanced use section of their docs). This is the most secure option if you know which cert the server should present.
>> 
>> Daniel Holth
>> 
>> On Jul 26, 2012, at 9:34 PM, Richard Jones <r1chardj0n3s at gmail.com> wrote:
>> 
>>> On 27 July 2012 11:17, Richard Jones <r1chardj0n3s at gmail.com> wrote:
>>>> Note that you need to use my branch of requests for this to work
>>>> (https://github.com/r1chardj0n3s/requests) and rauth currently needs
>>>> to be modified to handle unverifiable SSL certificates.
>>> 
>>> The author of rauth has shown me how to avoid the modification; the
>>> code on the wiki page now works with the current un-modified rauth
>>> release.
>>> 
>>> 
>>>   Richard
>>> _______________________________________________
>>> Catalog-SIG mailing list
>>> Catalog-SIG at python.org
>>> http://mail.python.org/mailman/listinfo/catalog-sig

From terence at honles.com  Sat Jul 28 22:47:19 2012
From: terence at honles.com (Terence Honles)
Date: Sat, 28 Jul 2012 13:47:19 -0700
Subject: [Catalog-sig] error adding/removing ssh keys
Message-ID: <CAHtY7_o5-ruJ3v1kvNeMujxdg+KvzinOMCM-CvAUQ-+J02Jd5g@mail.gmail.com>

When adding or removing SSH keys to my account I have been getting the
following error:

Content-Type: text/plain; charset=utf-8


> Error processing form


> Key processing failed. Please contact the administrator. Detail: Traceback
> (most recent call last):
>   File "/data/pypi/src/pypi/sshkeys_update.py", line 4, in <module>
>     c = config.Config("config.ini")
>   File "/data/pypi/src/pypi/config.py", line 10, in __init__
>     self.database_name = c.get('database', 'name')
>   File "/usr/lib/python2.6/ConfigParser.py", line 532, in get
>     raise NoSectionError(section)
> ConfigParser.NoSectionError: No section: 'database'


I do see the key in my user info, but when I ssh (pypissh) I get a public
key denied.

I have done the following:

   - ``ssh-add`` (with a valid agent running)
   - ``python -m pypissh``
   and
   ``ssh -T -l submit ssh.pypi.python.org``


-Terence
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20120728/c71c9aa9/attachment.html>

From aclark at aclark.net  Tue Jul 31 17:52:22 2012
From: aclark at aclark.net (Alex Clark)
Date: Tue, 31 Jul 2012 11:52:22 -0400
Subject: [Catalog-sig] ANN: pythonpackages.com beta
In-Reply-To: <20120731142402.GL10379@ragnarok.teratorn.org>
References: <jv1t98$8qr$2@dough.gmane.org>
	<20120730163118.GH10379@ragnarok.teratorn.org>
	<jv6dvl$2ih$1@dough.gmane.org>
	<20120730204904.GJ10379@ragnarok.teratorn.org>
	<jv6t5j$2vj$1@dough.gmane.org>
	<20120731142402.GL10379@ragnarok.teratorn.org>
Message-ID: <jv8uvq$bb3$1@dough.gmane.org>

Hi Eric,


(Continuing this discussion from twisted list)


On 7/31/12 10:24 AM, Eric P. Mangold wrote:
> On Mon, Jul 30, 2012 at 05:09:07PM -0400, Alex Clark wrote:
>> Hi Eric,
>>
>> On 7/30/12 4:49 PM, Eric P. Mangold wrote:
>>> On Mon, Jul 30, 2012 at 12:49:56PM -0400, Alex Clark wrote:
>>>> Hi,
>>>>
>>>>
>>>> On 7/30/12 12:31 PM, Eric P. Mangold wrote:
>>>>> Alex,
>>>>>
>>>>> I'm not sure if this is borderline off-topic, or not... but anyway..
>>>>>
>>>>> I'm sure starting a discussion here IS offtopic.
>>>>>
>>>>> But I have one question:
>>>>>
>>>>> How do package authors verify the integrity of their packages built "through the web"?
>>>>
>>>>
>>>> Good question, I just created:
>>>>
>>>> -
>>>> http://docs.pythonpackages.com/en/latest/faq.html#how-do-package-authors-verify-the-integrity-of-packages-built-through-the-web
>>>
>>> Let me be clear:
>>>
>>> Is it possible to have any assurance that your system has faithfully built the package, and/or that your servers have not been compromised?
>>>
>>> Why would anyone trust your web service to build packages, when it is *their* pgp, reputation and users that are at stake?
>>> (Yes, I would ask Launchpad/Canonical, et. all the same question...)
>>>
>>> (Also, if you're suggesting MD5 (following your link..) for anything related to security or data authenticity, then I *know* you're way off base.......)
>>
>>
>> The point about md5 is not to suggest using it for security or data
>> authenticity,
>
> Sorry, I think I have a problem with taking the exact text of my question,
> on your wiki, and casting it to be a different question entirely. (through
> no fault of your own, I'm sure)


Sorry, removed! Let me know if there is something better I can put in 
its place.


>
> I think I've clarified what my orignal question was meant to ask, namely how do
> we trust YOU and YOUR build infrastructure, not "how do we verify that the data
> you're give us hasn't been damaged in transit".
>
> If you wouldn't mind editing your wiki to reflect my clarifications, I would
> very much appreciate it :)


OK Let me work on it.


>
>> it's to clarify that whatever security is currently place
>> with PyPI (not a lot, admittedly) still applies, for whatever that is
>> worth (not much, apparently).
>>
>>
>>>
>>> Sorry if this is harsh - but it's intended. Without any kind of verifiable guarantee (get to work on that! :)) I don't think I could ever possibly use such a thing, and would advise against it.
>>>
>>> Getting software to end-users is a tough challenge, and I applaude your efforts to try and make it easier. A system with a single point of failure and a single point of trust just isn't feasible or desirable, imho.Administrators need to know who has final responsibility and *authority*
>> over the software that they are consuming. If "the cloud" is the last
>> link in that chain, then you have a big problem, I think.
>>
>>
>> The last link in the chain is PyPI (or a private index). The node before
>> that is typically your laptop. I'm suggesting you make it
>> pythonpackages.com instead.
>
> Clearly PyPI is inadequate. Jumping on solutions, for HARD problems that always
> require some form of cryptography to solve, is no more palettable.
>
> And PyPI is also just a publishing platform - the packages have already been
> minted by that point.
>
> So as you suggest, the last point is the developer/release-manager, as it should
> be.
>
> I think my point is that ideally you don't want to trust anyone except the
> developer/package-maintainer/release-manager.
>
> Debian et. all solve this with signed packages. I would be happy to download
> Debian packages from http://pythonpackages.com all day long :)


That's good to know, and probably I direction I'd like to head in. To be 
clear: I want to do any-useful-thing-I-can (within the ballpark) in 
order to start alleviating pain points for folks today.


>
> Debian also rely upon trusted build machines. But they are a more-or-less open
> organization with open review of what goes on.
>
> That said, I don't have a problem with people placing their trust in you. I don't
> know you, and don't have any opinion on it to be honest. You're probably a good guy ;)
>
> I would suggest working toward BEING a better PyPI mirror. Build
> the infrastructure necessary for people to publish python SOURCE packages,
> as they are, to PyPI, to pythonpackages.com, etc. etc. There is a lot of value
> to be added there.


Actually I'm mostly relying on the crate.io project (Donald Stufft) for 
this. I don't want pythonpackages.com to be a PyPI mirror, because other 
people are already doing this. The only related feature I'm considering 
(because folks have asked for it) is private PyPIs (something like 
index.pythonpackages.com only persistent).


>
> Build tools to make python packaging easy. On your laptop. On the cloud. Wherever.
> Open SOURCE is good like that.



Indeed! Currently working on a Windows version of pythonpackages.com to 
build Windows binaries (currently it only builds on Ubuntu).



Alex





>
> Regards,
> Eric Mangold
>


-- 
Alex Clark ? http://pythonpackages.com/ONE_CLICK


From dholth at gmail.com  Tue Jul 31 19:43:42 2012
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 31 Jul 2012 13:43:42 -0400
Subject: [Catalog-sig] ANN: pythonpackages.com beta
In-Reply-To: <jv8uvq$bb3$1@dough.gmane.org>
References: <jv1t98$8qr$2@dough.gmane.org>
	<20120730163118.GH10379@ragnarok.teratorn.org>
	<jv6dvl$2ih$1@dough.gmane.org>
	<20120730204904.GJ10379@ragnarok.teratorn.org>
	<jv6t5j$2vj$1@dough.gmane.org>
	<20120731142402.GL10379@ragnarok.teratorn.org>
	<jv8uvq$bb3$1@dough.gmane.org>
Message-ID: <CAG8k2+6GKZE5DaZ-uDLMN77ay0iLQ=H64J3O4eqLRuGxrqrW6w@mail.gmail.com>

Perhaps you would be interested in the Wheel package format's upcoming
public key signature system (wheel.rtfd.orgl#signed-wheel-files). The
(undocumented) plan will include per-buildserver, per-package and
possibly per-package-version signing keys via a pluggable trust model,
instead of the PGP model where a signing key is an e-mail address.

About wheel

"A wheel is a ZIP-format archive with a specially formatted filename
and the .whl extension. It is designed to contain all the files for a
PEP 376 compatible install in a way that is very close to the on-disk
format."

A simple wheel for Package-1.0 would contain

package.py

Package-1.0.dist-info/METADATA (PEP-376, PEP-345 'Metadata 1.2')

Package-1.0.dist-info/WHEEL (metadata for this build of the dist)

Package-1.0.dist-info/RECORD (extended PEP-376)

The bdist_wheel setuptools plugin and egg2wheel and wininst2wheel
allow you to create wheel archives without having to modify source
dists.

From eric at teratorn.org  Tue Jul 31 16:24:02 2012
From: eric at teratorn.org (Eric P. Mangold)
Date: Tue, 31 Jul 2012 10:24:02 -0400
Subject: [Catalog-sig] [Twisted-Python] ANN: pythonpackages.com beta
In-Reply-To: <jv6t5j$2vj$1@dough.gmane.org>
References: <jv1t98$8qr$2@dough.gmane.org>
	<20120730163118.GH10379@ragnarok.teratorn.org>
	<jv6dvl$2ih$1@dough.gmane.org>
	<20120730204904.GJ10379@ragnarok.teratorn.org>
	<jv6t5j$2vj$1@dough.gmane.org>
Message-ID: <20120731142402.GL10379@ragnarok.teratorn.org>

On Mon, Jul 30, 2012 at 05:09:07PM -0400, Alex Clark wrote:
> Hi Eric,
> 
> On 7/30/12 4:49 PM, Eric P. Mangold wrote:
> > On Mon, Jul 30, 2012 at 12:49:56PM -0400, Alex Clark wrote:
> >> Hi,
> >>
> >>
> >> On 7/30/12 12:31 PM, Eric P. Mangold wrote:
> >>> Alex,
> >>>
> >>> I'm not sure if this is borderline off-topic, or not... but anyway..
> >>>
> >>> I'm sure starting a discussion here IS offtopic.
> >>>
> >>> But I have one question:
> >>>
> >>> How do package authors verify the integrity of their packages built "through the web"?
> >>
> >>
> >> Good question, I just created:
> >>
> >> -
> >> http://docs.pythonpackages.com/en/latest/faq.html#how-do-package-authors-verify-the-integrity-of-packages-built-through-the-web
> >
> > Let me be clear:
> >
> > Is it possible to have any assurance that your system has faithfully built the package, and/or that your servers have not been compromised?
> >
> > Why would anyone trust your web service to build packages, when it is *their* pgp, reputation and users that are at stake?
> > (Yes, I would ask Launchpad/Canonical, et. all the same question...)
> >
> > (Also, if you're suggesting MD5 (following your link..) for anything related to security or data authenticity, then I *know* you're way off base.......)
> 
> 
> The point about md5 is not to suggest using it for security or data 
> authenticity,

Sorry, I think I have a problem with taking the exact text of my question,
on your wiki, and casting it to be a different question entirely. (through
no fault of your own, I'm sure)

I think I've clarified what my orignal question was meant to ask, namely how do
we trust YOU and YOUR build infrastructure, not "how do we verify that the data
you're give us hasn't been damaged in transit".

If you wouldn't mind editing your wiki to reflect my clarifications, I would
very much appreciate it :)

> it's to clarify that whatever security is currently place 
> with PyPI (not a lot, admittedly) still applies, for whatever that is 
> worth (not much, apparently).
> 
> 
> >
> > Sorry if this is harsh - but it's intended. Without any kind of verifiable guarantee (get to work on that! :)) I don't think I could ever possibly use such a thing, and would advise against it.
> >
> > Getting software to end-users is a tough challenge, and I applaude your efforts to try and make it easier. A system with a single point of failure and a single point of trust just isn't feasible or desirable, imho.Administrators need to know who has final responsibility and *authority* 
> over the software that they are consuming. If "the cloud" is the last 
> link in that chain, then you have a big problem, I think.
> 
> 
> The last link in the chain is PyPI (or a private index). The node before 
> that is typically your laptop. I'm suggesting you make it 
> pythonpackages.com instead.
 
Clearly PyPI is inadequate. Jumping on solutions, for HARD problems that always 
require some form of cryptography to solve, is no more palettable.

And PyPI is also just a publishing platform - the packages have already been
minted by that point.

So as you suggest, the last point is the developer/release-manager, as it should
be.

I think my point is that ideally you don't want to trust anyone except the
developer/package-maintainer/release-manager.

Debian et. all solve this with signed packages. I would be happy to download
Debian packages from http://pythonpackages.com all day long :)

Debian also rely upon trusted build machines. But they are a more-or-less open
organization with open review of what goes on.

That said, I don't have a problem with people placing their trust in you. I don't
know you, and don't have any opinion on it to be honest. You're probably a good guy ;)

I would suggest working toward BEING a better PyPI mirror. Build
the infrastructure necessary for people to publish python SOURCE packages,
as they are, to PyPI, to pythonpackages.com, etc. etc. There is a lot of value
to be added there.

Build tools to make python packaging easy. On your laptop. On the cloud. Wherever.
Open SOURCE is good like that.

Regards,
Eric Mangold