From martin at v.loewis.de  Mon Jan  4 00:04:15 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 04 Jan 2010 00:04:15 +0100
Subject: [Catalog-sig] PyPI OpenID changes
Message-ID: <4B41226F.8050805@v.loewis.de>

I have now changed PyPI so that it allows users to
type in their OpenID, for login, registration, and
association with an existing account.

If you find any problems, please submit a bug report.

Regards,
Martin

From ben+python at benfinney.id.au  Mon Jan  4 00:36:14 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 04 Jan 2010 10:36:14 +1100
Subject: [Catalog-sig] PyPI OpenID changes
References: <4B41226F.8050805@v.loewis.de>
Message-ID: <87hbr2zrpt.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> I have now changed PyPI so that it allows users to type in their
> OpenID, for login, registration, and association with an existing
> account.

Thank you for working on this.

> If you find any problems, please submit a bug report.

Unfortunately the bug tracker also doesn't support OpenID auth :-) I've
sent a report to you privately, I hope that's okay.

-- 
 \          ?One bad programmer can easily create two new jobs a year. |
  `\      Hiring more bad programmers will just increase our perceived |
_o__)                     need for them.? ?David Lorge Parnas, 1999-03 |
Ben Finney


From martin at v.loewis.de  Mon Jan  4 00:47:40 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 04 Jan 2010 00:47:40 +0100
Subject: [Catalog-sig] PyPI OpenID changes
In-Reply-To: <87hbr2zrpt.fsf@benfinney.id.au>
References: <4B41226F.8050805@v.loewis.de> <87hbr2zrpt.fsf@benfinney.id.au>
Message-ID: <4B412C9C.7030807@v.loewis.de>

>> If you find any problems, please submit a bug report.
> 
> Unfortunately the bug tracker also doesn't support OpenID auth :-) I've
> sent a report to you privately, I hope that's okay.

That's not true, it does - make sure to use the PyPI bug tracker.

Regards,
Martin

From ben+python at benfinney.id.au  Mon Jan  4 01:14:50 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 04 Jan 2010 11:14:50 +1100
Subject: [Catalog-sig] PyPI OpenID changes
References: <4B41226F.8050805@v.loewis.de> <87hbr2zrpt.fsf@benfinney.id.au>
Message-ID: <87d41qzpxh.fsf@benfinney.id.au>

Ben Finney <ben+python at benfinney.id.au> writes:

> "Martin v. L?wis" <martin at v.loewis.de> writes:
> > If you find any problems, please submit a bug report.
>
> Unfortunately the bug tracker also doesn't support OpenID auth :-)

As Martin points out, the bug tracker for PyPI is not the Python bug
tracker.

I usually find it's best to give a URL to the bug tracker in the same
message asking people to submit to it. Here is the URL for the PyPI bug
tracker <URL:http://sourceforge.net/tracker/?group_id=66150>.

-- 
 \                ?Every sentence I utter must be understood not as an |
  `\                      affirmation, but as a question.? ?Niels Bohr |
_o__)                                                                  |
Ben Finney


From ben+python at benfinney.id.au  Mon Jan  4 01:57:04 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 04 Jan 2010 11:57:04 +1100
Subject: [Catalog-sig] PyPI OpenID changes
References: <4B41226F.8050805@v.loewis.de> <87hbr2zrpt.fsf@benfinney.id.au>
	<87d41qzpxh.fsf@benfinney.id.au>
Message-ID: <878wceznz3.fsf@benfinney.id.au>

Ben Finney <ben+python at benfinney.id.au> writes:

> I usually find it's best to give a URL to the bug tracker in the same
> message asking people to submit to it. Here is the URL for the PyPI
> bug tracker <URL:http://sourceforge.net/tracker/?group_id=66150>.

Good luck to those trying to use it. For me, the bug tracker login
<URL:https://sourceforge.net/account/login.php> only leads to a timeout.
I'll have to resort to email submission of bug reports.

-- 
 \       ?Crime is contagious? if the government becomes a lawbreaker, |
  `\          it breeds contempt for the law.? ?Justice Louis Brandeis |
_o__)                                                                  |
Ben Finney


From martin at v.loewis.de  Mon Jan  4 09:11:30 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 04 Jan 2010 09:11:30 +0100
Subject: [Catalog-sig] PyPI OpenID changes
In-Reply-To: <878wceznz3.fsf@benfinney.id.au>
References: <4B41226F.8050805@v.loewis.de>
	<87hbr2zrpt.fsf@benfinney.id.au>	<87d41qzpxh.fsf@benfinney.id.au>
	<878wceznz3.fsf@benfinney.id.au>
Message-ID: <4B41A2B2.5000907@v.loewis.de>

>> I usually find it's best to give a URL to the bug tracker in the same
>> message asking people to submit to it. Here is the URL for the PyPI
>> bug tracker <URL:http://sourceforge.net/tracker/?group_id=66150>.
> 
> Good luck to those trying to use it. For me, the bug tracker login
> <URL:https://sourceforge.net/account/login.php> only leads to a timeout.
> I'll have to resort to email submission of bug reports.

Please understand that chances are high that I ignore/forget about
email bug reports.

Regards,
Martin


From ben+python at benfinney.id.au  Mon Jan  4 10:51:27 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 04 Jan 2010 20:51:27 +1100
Subject: [Catalog-sig] PyPI OpenID changes
References: <4B41226F.8050805@v.loewis.de> <87hbr2zrpt.fsf@benfinney.id.au>
	<87d41qzpxh.fsf@benfinney.id.au> <878wceznz3.fsf@benfinney.id.au>
	<4B41A2B2.5000907@v.loewis.de>
Message-ID: <87k4vyxko0.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> > Good luck to those trying to use [the Sourceforge bug tracker]. For
> > me, the bug tracker login
> > <URL:https://sourceforge.net/account/login.php> only leads to a
> > timeout. I'll have to resort to email submission of bug reports.
>
> Please understand that chances are high that I ignore/forget about
> email bug reports.

Understood.

You might want to solicit assistance (if you know what assistance is
needed) to switch away from the Sourceforge bug tracker, if you feel
that there are parallels between Python's reasons for switching away and
PyPI.

-- 
 \       ?Two possibilities exist: Either we are alone in the Universe |
  `\   or we are not. Both are equally terrifying.? ?Arthur C. Clarke, |
_o__)                                                             1999 |
Ben Finney


From martin at v.loewis.de  Mon Jan  4 22:24:59 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 04 Jan 2010 22:24:59 +0100
Subject: [Catalog-sig] PyPI OpenID changes
In-Reply-To: <87k4vyxko0.fsf@benfinney.id.au>
References: <4B41226F.8050805@v.loewis.de>
	<87hbr2zrpt.fsf@benfinney.id.au>	<87d41qzpxh.fsf@benfinney.id.au>
	<878wceznz3.fsf@benfinney.id.au>	<4B41A2B2.5000907@v.loewis.de>
	<87k4vyxko0.fsf@benfinney.id.au>
Message-ID: <4B425CAB.6020100@v.loewis.de>

>>> Good luck to those trying to use [the Sourceforge bug tracker]. For
>>> me, the bug tracker login
>>> <URL:https://sourceforge.net/account/login.php> only leads to a
>>> timeout. I'll have to resort to email submission of bug reports.
>> Please understand that chances are high that I ignore/forget about
>> email bug reports.
> 
> Understood.
> 
> You might want to solicit assistance (if you know what assistance is
> needed) to switch away from the Sourceforge bug tracker, if you feel
> that there are parallels between Python's reasons for switching away and
> PyPI.

There may be reasons to switch to a different bug tracker, but not
parallels. For example, one report in the tracker asked for a bug
tracker that accepts the same passwords as PyPI. OTOH, other parallels
may not apply: e.g. it doesn't need to be written in Python.

So: if anybody would volunteer to setup a bug tracker for PyPI that
accepts the same passwords as PyPI (and probably should also support
OpenID, preferably with the same IDs associated with users as in PyPI),
please step forward. Moving to the new bug tracker would preferably
also move all open reports (closed reports optional).

Regards,
Martin

From jmg3000 at gmail.com  Thu Jan  7 16:12:01 2010
From: jmg3000 at gmail.com (John Gabriele)
Date: Thu, 7 Jan 2010 10:12:01 -0500
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B45C9F8.6070608@trueblade.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
Message-ID: <65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>

On Thu, Jan 7, 2010 at 9:56 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Thu, Jan 7, 2010 at 3:52 PM, John Gabriele <jmg3000 at gmail.com> wrote:
>> The only inconsistency, I think, is that operating systems like Debian
>> refer to their software distributions as "packages" (as in, a packaged
>> up piece of software that you can download and install). "Packages" is
>> a great name for them -- too bad it's already being used in Python.
>
> I believe that's basically where the confusion comes from.

Whoops. Just noticed that the front page of the PyPI says:

> "There are currently 8614 packages here."

(is that 8614 packages or 8614 distributions?)

and,

> "To submit a package use "python setup.py upload" and to use a package from this index either "pip install package" or download, unpack and "python setup.py install" it."

and

> "# Browse the tree of packages
# Submit package information"

---John

From pje at telecommunity.com  Thu Jan  7 17:44:24 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 07 Jan 2010 11:44:24 -0500
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.co
 m>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B45C9F8.6070608@trueblade.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
Message-ID: <20100107164433.662DE3A4074@sparrow.telecommunity.com>

At 10:29 AM 1/7/2010 -0600, Brad Allen wrote:
>On Thu, Jan 7, 2010 at 9:12 AM, John Gabriele <jmg3000 at gmail.com> wrote:
> > On Thu, Jan 7, 2010 at 9:56 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> >> On Thu, Jan 7, 2010 at 3:52 PM, John Gabriele <jmg3000 at gmail.com> wrote:
> >>> The only inconsistency, I think, is that operating systems like Debian
> >>> refer to their software distributions as "packages" (as in, a packaged
> >>> up piece of software that you can download and install). "Packages" is
> >>> a great name for them -- too bad it's already being used in Python.
> >>
> >> I believe that's basically where the confusion comes from.
> >
> > Whoops. Just noticed that the front page of the PyPI says:
> >
> >> "There are currently 8614 packages here."
> >
> > (is that 8614 packages or 8614 distributions?)

8614 *projects*, some of which have one or more *versions*, which in 
turn may have one or more source or binary *distributions*.  That at 
least is the terminology that setuptools and distribute use in their 
documentation at the moment.


From bradallen137 at gmail.com  Thu Jan  7 17:29:45 2010
From: bradallen137 at gmail.com (Brad Allen)
Date: Thu, 7 Jan 2010 10:29:45 -0600
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B45C9F8.6070608@trueblade.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
Message-ID: <4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>

On Thu, Jan 7, 2010 at 9:12 AM, John Gabriele <jmg3000 at gmail.com> wrote:
> On Thu, Jan 7, 2010 at 9:56 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> On Thu, Jan 7, 2010 at 3:52 PM, John Gabriele <jmg3000 at gmail.com> wrote:
>>> The only inconsistency, I think, is that operating systems like Debian
>>> refer to their software distributions as "packages" (as in, a packaged
>>> up piece of software that you can download and install). "Packages" is
>>> a great name for them -- too bad it's already being used in Python.
>>
>> I believe that's basically where the confusion comes from.
>
> Whoops. Just noticed that the front page of the PyPI says:
>
>> "There are currently 8614 packages here."
>
> (is that 8614 packages or 8614 distributions?)
>
> and,
>
>> "To submit a package use "python setup.py upload" and to use a package from this index either "pip install package" or download, unpack and "python setup.py install" it."
>
> and
>
>> "# Browse the tree of packages
> # Submit package information"

Right! The word 'package' is used all over the place, including the
name PyPI abbreviates (Python Package Index), and the title of
PEP-345. Haven't we all also encountered this common usage? I've
noticed in various discussions on this mailing list.

Furthermore, even the official docs make an apology about this
terminology that no one seems to use, stating that the word 'package'
would be preferred but that term is already taken:

From:
> http://docs.python.org/distutils/introduction.html#distutils-specific-terminology

"This would be called a package, except that term is already taken in
the Python context: a single module distribution may contain zero,
one, or many Python packages."

If the official terminology has not caught on in common parlance, is
it really worth maintaining? The word 'package' seems to be what
people want to use, and it is a more accessible term to newbies,
system package maintainers, etc. Rather than waste energy fighting
this popular usage, why not find a solution for this little namespace
conflict?

Of many possible solutions; here are some ideas:

* Make the word 'package' an official replacement for
'module-distribution', without even deprecating the old 'package'
usage for __init__.py directories. English is full of words with
contextual meaning, and people can usually tell by the context which
kind of package is being referred to. After all, that's what's is
happening today. There could be a document clearly stating the
different kinds of packages.

* Use a qualifier, like one of these: 'PyPI Package', 'Setup Package',
'Project Package' while the old kind of package could be qualified as
'Python package' or 'Module Package'.

* Deprecate the old use of 'package' and replace it with another word
such as 'bundle', and try to get it into all Python 3 documentation.

* Introduce a catchy new term to quell the popular tendency to use the
word package instead of 'module-distribution'. Ideas: 'MetaPackage',
'Pkg', 'Porridge', 'Punkage', 'Spankage', 'Spackage', 'Pancake',

From martin at v.loewis.de  Thu Jan  7 21:20:23 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 07 Jan 2010 21:20:23 +0100
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <20100107164433.662DE3A4074@sparrow.telecommunity.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>	<4B45C9F8.6070608@trueblade.com>	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
Message-ID: <4B464207.3030507@v.loewis.de>

> 8614 *projects*, some of which have one or more *versions*, which in
> turn may have one or more source or binary *distributions*.

Instead of "version", I really like PyPI's term more: *releases*.
As for projects: fine with me; PyPI would then be the Python Project
Index.

Regards,
Martin

From pje at telecommunity.com  Thu Jan  7 21:40:53 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 07 Jan 2010 15:40:53 -0500
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <4B464207.3030507@v.loewis.de>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B45C9F8.6070608@trueblade.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
Message-ID: <20100107204102.D121C3A4074@sparrow.telecommunity.com>

At 09:20 PM 1/7/2010 +0100, Martin v. L?wis wrote:
> > 8614 *projects*, some of which have one or more *versions*, which in
> > turn may have one or more source or binary *distributions*.
>
>Instead of "version", I really like PyPI's term more: *releases*.

Not all versions are released versions, so I suppose it's actually:

project -> version -> release -> distribution


>As for projects: fine with me; PyPI would then be the Python Project
>Index.

+1.


From g.brandl at gmx.net  Thu Jan  7 21:57:33 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 07 Jan 2010 21:57:33 +0100
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <4B464207.3030507@v.loewis.de>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>	<4B45C9F8.6070608@trueblade.com>	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
Message-ID: <hi5hsl$7ev$1@ger.gmane.org>

Am 07.01.2010 21:20, schrieb "Martin v. L?wis":
>> 8614 *projects*, some of which have one or more *versions*, which in
>> turn may have one or more source or binary *distributions*.
> 
> Instead of "version", I really like PyPI's term more: *releases*.
> As for projects: fine with me; PyPI would then be the Python Project
> Index.

"It's just a different kind of cheese..."

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From martin at v.loewis.de  Thu Jan  7 22:00:33 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 07 Jan 2010 22:00:33 +0100
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <20100107204102.D121C3A4074@sparrow.telecommunity.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B45C9F8.6070608@trueblade.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
Message-ID: <4B464B71.9020003@v.loewis.de>

>> Instead of "version", I really like PyPI's term more: *releases*.
> 
> Not all versions are released versions

Actually, from a PyPI point of view, they are :-)

Regards,
Martin

From ziade.tarek at gmail.com  Thu Jan  7 22:39:33 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 7 Jan 2010 22:39:33 +0100
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
Message-ID: <94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>

On Thu, Jan 7, 2010 at 10:29 PM, Brad Allen <bradallen137 at gmail.com> wrote:
> On Thu, Jan 7, 2010 at 2:40 PM, P.J. Eby <pje at telecommunity.com> wrote:
>
>>> As for projects: fine with me; PyPI would then be the Python Project
>>> Index.
>
> +1
>
> If this gets general agreement, there are probably some places where
> the word 'package' should be replaced with the word 'project', right?
> For example, should PEP-345 be retitled as "Metadata for Python
> Software Projects 1.2" ?

+1, we should reflect this terminology in all new PEPs and in
Distutils doc as well

> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziad? | http://ziade.org

From mal at egenix.com  Thu Jan  7 22:51:43 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 07 Jan 2010 22:51:43 +0100
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>	<20100107164433.662DE3A4074@sparrow.telecommunity.com>	<4B464207.3030507@v.loewis.de>	<20100107204102.D121C3A4074@sparrow.telecommunity.com>	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
Message-ID: <4B46576F.7070309@egenix.com>

Tarek Ziad? wrote:
> On Thu, Jan 7, 2010 at 10:29 PM, Brad Allen <bradallen137 at gmail.com> wrote:
>> On Thu, Jan 7, 2010 at 2:40 PM, P.J. Eby <pje at telecommunity.com> wrote:
>>
>>>> As for projects: fine with me; PyPI would then be the Python Project
>>>> Index.
>>
>> +1
>>
>> If this gets general agreement, there are probably some places where
>> the word 'package' should be replaced with the word 'project', right?
>> For example, should PEP-345 be retitled as "Metadata for Python
>> Software Projects 1.2" ?
> 
> +1, we should reflect this terminology in all new PEPs and in
> Distutils doc as well

I don't think we need to change anything - most Python software
components come as Python packages nowadays, so the terminology
'package' we've used all these years is correct.

OTOH, sets of components like Zope are indeed projects. However,
the number of PyPI packages which could reasonably be called a project
is relatively small and even those use Python packages to
separate their namespaces.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 07 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From stephen.c.waterbury at nasa.gov  Thu Jan  7 22:47:43 2010
From: stephen.c.waterbury at nasa.gov (Stephen Waterbury)
Date: Thu, 7 Jan 2010 16:47:43 -0500
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>	<4B45C9F8.6070608@trueblade.com>	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>	<20100107164433.662DE3A4074@sparrow.telecommunity.com>	<4B464207.3030507@v.loewis.de>	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
Message-ID: <4B46567F.5070606@nasa.gov>

Brad Allen wrote:
> On Thu, Jan 7, 2010 at 2:40 PM, P.J. Eby <pje at telecommunity.com> wrote:
> 
>>> As for projects: fine with me; PyPI would then be the Python Project
>>> Index.
> 
> +1
> 
> If this gets general agreement, there are probably some places where
> the word 'package' should be replaced with the word 'project', right?
> For example, should PEP-345 be retitled as "Metadata for Python
> Software Projects 1.2" ?

+1

This makes a huge amount of sense, both from a semantic point of view --
"projects" with "releases" seems a better fit for what is in PyPI --
and as a resolution to the name collision with the Python "package" idiom.

Steve

From mal at egenix.com  Thu Jan  7 23:12:47 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 07 Jan 2010 23:12:47 +0100
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>	<20100107164433.662DE3A4074@sparrow.telecommunity.com>	<4B464207.3030507@v.loewis.de>	<20100107204102.D121C3A4074@sparrow.telecommunity.com>	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
Message-ID: <4B465C5F.7030609@egenix.com>

Brad Allen wrote:
> On Thu, Jan 7, 2010 at 3:51 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> I don't think we need to change anything - most Python software
>> components come as Python packages nowadays, so the terminology
>> 'package' we've used all these years is correct.
> 
> Do you mean only 'package' in the sense of an __init__.py container,
> or in the sense of a setup.py container? Both usages have been in use
> for years, but only the __init__.py package was formally recognized.

We have used the term 'package' for both a Python
software component as well as the Python module directories on
sys.path with a __init__.py marker.

I usually refer to the latter as 'Python package' and the former
as 'package'.

What you download is a 'distribution file' which could be an exe,
a tarball, an egg, etc. There are many forms such a package
distribution can take and just as many ways to install them.

Once installed a 'package' usually creates a single 'Python package'
(at top-level or some lower level).

As a result, after installation there is little difference between
a Python software component called 'package' and the module
directory called 'Python package'. qed :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 07 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Thu Jan  7 23:22:52 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 7 Jan 2010 23:22:52 +0100
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <4B465C5F.7030609@egenix.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
Message-ID: <94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>

On Thu, Jan 7, 2010 at 11:12 PM, M.-A. Lemburg <mal at egenix.com> wrote:
[..]
> I usually refer to the latter as 'Python package' and the former
> as 'package'.
>
> What you download is a 'distribution file' which could be an exe,
> a tarball, an egg, etc. There are many forms such a package
> distribution can take and just as many ways to install them.

That would be a "release" instead imho. (I like Tres' definitions for that)
e.g. a "release" comes in several "distributions" :  an egg, a source
dist, a binary dist, etc.

>
> Once installed a 'package' usually creates a single 'Python package'
> (at top-level or some lower level).

That's not always the case. Many projects have also scripts, and /or
top level modules in site-packages.

Cheers
Tarek

From pje at telecommunity.com  Fri Jan  8 00:19:28 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 07 Jan 2010 18:19:28 -0500
Subject: [Catalog-sig] [Distutils]   packaging terminology confusion
In-Reply-To: <4B46576F.7070309@egenix.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
Message-ID: <20100107231937.8F1293A4108@sparrow.telecommunity.com>

At 10:51 PM 1/7/2010 +0100, M.-A. Lemburg wrote:
>I don't think we need to change anything - most Python software
>components come as Python packages nowadays, so the terminology
>'package' we've used all these years is correct.

Actually, this sentence rather proves the *opposite* point, since 
Brad had to ask which meaning you intended for "packages" in it.  ;-)

(And I confess I was a bit confused myself, since it could be read 
either as a tautology or a dubious proposition, depending on which 
way one took it.)


From fuzzyman at gmail.com  Fri Jan  8 00:52:53 2010
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 7 Jan 2010 23:52:53 +0000
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <20100107204102.D121C3A4074@sparrow.telecommunity.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B45C9F8.6070608@trueblade.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
Message-ID: <6f4025011001071552r980df04w23e3afb5a3efd71a@mail.gmail.com>

2010/1/7 P.J. Eby <pje at telecommunity.com>

> At 09:20 PM 1/7/2010 +0100, Martin v. L?wis wrote:
>
>> > 8614 *projects*, some of which have one or more *versions*, which in
>> > turn may have one or more source or binary *distributions*.
>>
>> Instead of "version", I really like PyPI's term more: *releases*.
>>
>
> Not all versions are released versions, so I suppose it's actually:
>
> project -> version -> release -> distribution
>
>
>
>  As for projects: fine with me; PyPI would then be the Python Project
>> Index.
>>
>
> +1.
>
>
+1 from me as well. It does remove ambiguity.

Michael



>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>



-- 
http://www.ironpythoninaction.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100107/24ed2c02/attachment-0001.htm>

From bradallen137 at gmail.com  Thu Jan  7 22:29:32 2010
From: bradallen137 at gmail.com (Brad Allen)
Date: Thu, 7 Jan 2010 15:29:32 -0600
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <20100107204102.D121C3A4074@sparrow.telecommunity.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B45C9F8.6070608@trueblade.com>
	<94bdd2611001070403u2834bdc3o87eab30dc5c8b7e4@mail.gmail.com>
	<65e0bb521001070652k6925da3ak3a89ca484234f6b5@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
Message-ID: <4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>

On Thu, Jan 7, 2010 at 2:40 PM, P.J. Eby <pje at telecommunity.com> wrote:

>> As for projects: fine with me; PyPI would then be the Python Project
>> Index.

+1

If this gets general agreement, there are probably some places where
the word 'package' should be replaced with the word 'project', right?
For example, should PEP-345 be retitled as "Metadata for Python
Software Projects 1.2" ?

From bradallen137 at gmail.com  Thu Jan  7 22:57:22 2010
From: bradallen137 at gmail.com (Brad Allen)
Date: Thu, 7 Jan 2010 15:57:22 -0600
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <4B46576F.7070309@egenix.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001070656k3268b2b9n8f5bc2fdab747f3d@mail.gmail.com>
	<65e0bb521001070712o28cd5d92mc46c4fd4fcb4fca@mail.gmail.com>
	<4957f1ef1001070829h70f0ecc8m25ddd314ccb1b415@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
Message-ID: <4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>

On Thu, Jan 7, 2010 at 3:51 PM, M.-A. Lemburg <mal at egenix.com> wrote:

> I don't think we need to change anything - most Python software
> components come as Python packages nowadays, so the terminology
> 'package' we've used all these years is correct.

Do you mean only 'package' in the sense of an __init__.py container,
or in the sense of a setup.py container? Both usages have been in use
for years, but only the __init__.py package was formally recognized.

From jmg3000 at gmail.com  Fri Jan  8 04:43:06 2010
From: jmg3000 at gmail.com (John Gabriele)
Date: Thu, 7 Jan 2010 22:43:06 -0500
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
Message-ID: <65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>

On Thu, Jan 7, 2010 at 5:22 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Thu, Jan 7, 2010 at 11:12 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> [..]
>> I usually refer to the latter as 'Python package' and the former
>> as 'package'.
>>
>> What you download is a 'distribution file' which could be an exe,
>> a tarball, an egg, etc. There are many forms such a package
>> distribution can take and just as many ways to install them.
>
> That would be a "release" instead imho. (I like Tres' definitions for that)
> e.g. a "release" comes in several "distributions" : ?an egg, a source
> dist, a binary dist, etc.
>
>>
>> Once installed a 'package' usually creates a single 'Python package'
>> (at top-level or some lower level).
>
> That's not always the case. Many projects have also scripts, and /or
> top level modules in site-packages.

I've got a few brief observations, and also a suggestion.

1. In english, the word "package" can mean an aggregation of features,
as in, "We got married, stayed at a hotel, and got the honeymoon
package. Heart-shaped bed, champagne, and chocolate-covered
strawberries in the fridge!". So, it's a great name for something that
aggregates things, like modules.

2. Also, in english, the word "package" can also mean a nice sealed
box that you send out via or receive from the postal mail. So, it's a
great name for those tgz things that you upload to or download from
the PyPI.

3. People don't like calling those MyProject-1.0.2.tgz thingies
"distributions". They keep calling them packages, and when you correct
them, they say, "[sigh], fine. [eyes roll] 'distribution'."

4. A "project" is just a general term for a named thing that people
work on. "I'm working on my Isengard Reforestation project! I think
it's going to be very successful! I'm keeping it in my
`~/dev/projects/IsengardReforestation` dir." But I don't think
`IsengardReforestation-1.0.2.tgz` would be referred to as a "project"
per se, nor a "project file". A "project file" is one of those things
that an IDE gives you to keep track of which files belong in the
project, how to build it, and so on---so I wouldn't call it that
either.

5. There are already a bunch of tools that refer to those PyPI tgz
things as distributions (for example, Distutils).

6. The word "distribution" is long to type and long to say. And also,
when I think of a Python distribution, I imagine the
`Python-2.6.4.tgz` file you download from python.org. *That's* a
Python distribution (along with the previous Python releases). Also,
ActiveState has a Python distribution too.

So, here's a suggestion: just call both things (packages and
distributions) "packages", but then agree to fully qualify the names
when you need to avoid ambiguity, for example: "I just built a
distribution-package (or "dist-package" for short) and included
numerous module-packages in it."

* If you mean the kind of package holds modules, and the meaning isn't
clear from the context, say "module-package" (think "honeymoon
package")

* If you mean the kind of package hosted at the PyPI, and the meaning
isn't clear from the context, say "distribution-package" or
"dist-package" (think parcel/postal package)

This way,

* The PyPI now becomes the "Python dist-package index", or just
"Python Package Index (PyPI)" for short. :)

* Distutils is now the "distribution-package utilities" or just "dist
utilities" for short. :)

* We can slowly fully-qualify the name "distribution" in the docs,
gradually replacing it with "distribution-package" or "dist-package"
(when it's referring to dist-packages) and it won't be much of a shock
to the system.

How's that sound?

---John

From martin at v.loewis.de  Fri Jan  8 04:58:43 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 08 Jan 2010 04:58:43 +0100
Subject: [Catalog-sig] [Distutils]   packaging terminology confusion
In-Reply-To: <65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>	<20100107164433.662DE3A4074@sparrow.telecommunity.com>	<4B464207.3030507@v.loewis.de>	<20100107204102.D121C3A4074@sparrow.telecommunity.com>	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>	<4B46576F.7070309@egenix.com>	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>	<4B465C5F.7030609@egenix.com>	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
Message-ID: <4B46AD73.4000902@v.loewis.de>

> How's that sound?

I read this as a -1 for changing PyPI to use "project".

Still, support *for* renaming it is much wider, it seems.

Regards,
Martin

From jmg3000 at gmail.com  Fri Jan  8 05:30:44 2010
From: jmg3000 at gmail.com (John Gabriele)
Date: Thu, 7 Jan 2010 23:30:44 -0500
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
In-Reply-To: <4B46AD73.4000902@v.loewis.de>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<4B46AD73.4000902@v.loewis.de>
Message-ID: <65e0bb521001072030t333ab75egac12cd916fa1bd93@mail.gmail.com>

On Thu, Jan 7, 2010 at 10:58 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> How's that sound?
>
> I read this as a -1 for changing PyPI to use "project".

Well, I'd say not -1, nor +1. I think either name for the PyPI is ok.

I was trying to address the larger issue of confusion of terminology.
I think that no matter what you call the site, in common use users
will still catch themselves saying "get a package from the PyPI",
rather than "get a project from the PyPI". I mean, even some experts
here accidentally refer to distributions as packages.

---John

From fdrake at gmail.com  Fri Jan  8 05:47:05 2010
From: fdrake at gmail.com (Fred Drake)
Date: Thu, 7 Jan 2010 23:47:05 -0500
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com> 
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com> 
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com> 
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com> 
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com> 
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com> 
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
Message-ID: <9cee7ab81001072047j40946dew9ee32f5231ed1d0@mail.gmail.com>

On Thu, Jan 7, 2010 at 10:43 PM, John Gabriele <jmg3000 at gmail.com> wrote:
> 3. People don't like calling those MyProject-1.0.2.tgz thingies
> "distributions". They keep calling them packages, and when you correct
> them, they say, "[sigh], fine. [eyes roll] 'distribution'."

Interesting.  I've never observed any evidence of that.  Yes, some
people call them packages.  Others call them distributions.  Even
outside the Python realm.

Of course, if you'd rather call them tarballs, I won't mind.  Even if
you don't use tar to make them.  :-)  Or they could be GBOFs (Great
Balls of Fire).

As for the "get a package from the PyPI" verbage, I don't think that's
an issue.  I suspect most distributions include packages rather than a
handful of separate modules these days.  Nor is it wrong to use a
looser term (still widely accepted in the broader community) for the
things gotten from PyPI.

The point of carefully defined terminology is primarily to make sure
that relatively formal communication, such as technical documentation,
can be carried out both effectively and efficiently.  There's no need
to dictate terminology for casual communication, where context is
usually sufficient.


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller

From jmg3000 at gmail.com  Fri Jan  8 05:47:56 2010
From: jmg3000 at gmail.com (John Gabriele)
Date: Thu, 7 Jan 2010 23:47:56 -0500
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
Message-ID: <65e0bb521001072047w3b5ef03aub50a926ebad47f16@mail.gmail.com>

On Thu, Jan 7, 2010 at 10:43 PM, John Gabriele <jmg3000 at gmail.com> wrote:
>
> So, here's a suggestion: just call both things (packages and
> distributions) "packages", but then agree to fully qualify the names
> when you need to avoid ambiguity, for example: "I just built a
> distribution-package (or "dist-package" for short) and included
> numerous module-packages in it."
>
> * If you mean the kind of package holds modules, and the meaning isn't
> clear from the context, say "module-package" (think "honeymoon
> package")

Whoops. Looking back through this thread, that last line should probably read:

"""
... say, as Brad A. suggested above, "module-package" ...
"""

---John

From tseaver at palladion.com  Fri Jan  8 07:13:43 2010
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 08 Jan 2010 01:13:43 -0500
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <9cee7ab81001072047j40946dew9ee32f5231ed1d0@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B464207.3030507@v.loewis.de>	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<9cee7ab81001072047j40946dew9ee32f5231ed1d0@mail.gmail.com>
Message-ID: <hi6iem$qag$2@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Fred Drake wrote:
> On Thu, Jan 7, 2010 at 10:43 PM, John Gabriele <jmg3000 at gmail.com> wrote:
>> 3. People don't like calling those MyProject-1.0.2.tgz thingies
>> "distributions". They keep calling them packages, and when you correct
>> them, they say, "[sigh], fine. [eyes roll] 'distribution'."
> 
> Interesting.  I've never observed any evidence of that.  Yes, some
> people call them packages.  Others call them distributions.  Even
> outside the Python realm.
> 
> Of course, if you'd rather call them tarballs, I won't mind.  Even if
> you don't use tar to make them.  :-)  Or they could be GBOFs (Great
> Balls of Fire).
> 
> As for the "get a package from the PyPI" verbage, I don't think that's
> an issue.  I suspect most distributions include packages rather than a
> handful of separate modules these days.  Nor is it wrong to use a
> looser term (still widely accepted in the broader community) for the
> things gotten from PyPI.
> 
> The point of carefully defined terminology is primarily to make sure
> that relatively formal communication, such as technical documentation,
> can be carried out both effectively and efficiently.  There's no need
> to dictate terminology for casual communication, where context is
> usually sufficient.

Amen!  Let's go for precision in PEPs, docs, and other spec-like stuff,
and not worry about correcting imprecise-but-easy-to-disambiguate-in-
context usage in ordinary discourse.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAktGzRcACgkQ+gerLs4ltQ77jQCgxjHfopLzY7b/s6mLYBtb8/d+
WcMAoM7zU+TScc+XFhrHKxF3W8hRNB1m
=sbQz
-----END PGP SIGNATURE-----


From jmg3000 at gmail.com  Fri Jan  8 16:40:41 2010
From: jmg3000 at gmail.com (John Gabriele)
Date: Fri, 8 Jan 2010 10:40:41 -0500
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
In-Reply-To: <723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
Message-ID: <65e0bb521001080740o4e09a0f1ib49e8ec555ec611c@mail.gmail.com>

On Fri, Jan 8, 2010 at 1:50 AM, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> On Jan 7, 2010, at 10:43 PM, John Gabriele wrote:
>
> 3. People don't like calling those MyProject-1.0.2.tgz thingies
> "distributions". They keep calling them packages, and when you correct
> them, they say, "[sigh], fine. [eyes roll] 'distribution'."
>
> This is not my experience of the term. ?It's more that most people *still*
> don't know what the term "distribution" means in this context, unless
> they've extensively read the distutils documentation. ?Or that they're
> talking about Python "distributions" at the same time as what Linux
> distributions are doing with them, and are trying to avoid being ambiguous.

Hm. Well, it's possible I'm reading too much into online commentary
while wearing my cheese-colored glasses.

> It sure would be nice if we could use a made-up word like "eggs" to refer to
> these things. ?Too bad that's taken too :-\.

If there's a desire to replace the word "distribution" (meaning, the
tgz's at the PyPI) in the technical docs to increase clarity, then
your suggestion of "parcel" is certainly good. Another might be
"bundle" (though, that one's used on Mac OS X in connection with
shared libs, I think).

I like saying "Python distribution" to mean the big archive file you
download to install Python on your system.

Of course, the PyPI could be renamed "Python Project Index" regardless
of the technical name used for Fred's GBOFs. :)

I also think that the instructions on the front page of the PyPI could
say "usage: `pip install ProjectName` or `python setup.py install
ProjectName`" and it would be clear, again, regardless of what the
technical name of GBOFs are.

Re. Barry's recent comment:
>
> I wish there was a humorous, ironic name for the service that is evocative of
> {snip}

Could always come up with such a name, and still use "The Python
Project Index" as the main *subtitle*.

---John

From bradallen137 at gmail.com  Sat Jan  9 16:23:56 2010
From: bradallen137 at gmail.com (Brad Allen)
Date: Sat, 9 Jan 2010 09:23:56 -0600
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <9cee7ab81001072047j40946dew9ee32f5231ed1d0@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<9cee7ab81001072047j40946dew9ee32f5231ed1d0@mail.gmail.com>
Message-ID: <4957f1ef1001090723m659e36cbv220b02e6285744b1@mail.gmail.com>

On Thu, Jan 7, 2010 at 10:47 PM, Fred Drake <fdrake at gmail.com> wrote:

> The point of carefully defined terminology is primarily to make sure
> that relatively formal communication, such as technical documentation,
> can be carried out both effectively and efficiently. ?There's no need
> to dictate terminology for casual communication, where context is
> usually sufficient.

Fred Drake said:
> The point of carefully defined terminology is primarily to make sure
> that relatively formal communication, such as technical documentation,
> can be carried out both effectively and efficiently.

Nevertheless, formal terminology imost useful when it agrees with
common usage; after all, we also want informal discussion (such as we
have on distutils, everyday work, user groups, etc.) to be effective
and efficient. In this case the formal terminology is so cumbersome
that people tend not to use in technical documentation.

>There's no need
> to dictate terminology for casual communication, where context is
> usually sufficient.

Ah, well we can't dictate that anyway, can we? This is a negotiation,
not an attempt to dictate. People will say what they want to say. The
goal here is to improve the formal terminology and change it in the
documentation, in such a way that informal discussions are more likely
to adopt the formal terminology and to be consistent with the
documentation.

From jmg3000 at gmail.com  Sun Jan 10 00:04:40 2010
From: jmg3000 at gmail.com (John Gabriele)
Date: Sat, 9 Jan 2010 18:04:40 -0500
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
In-Reply-To: <319e029f1001091148j3c6a929g6867a89efb5da776@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<319e029f1001080829w4a140dedr48e0d4dce249caed@mail.gmail.com>
	<4957f1ef1001080900h52f271ccg45f71f376592db79@mail.gmail.com>
	<20100108174506.B2C473A4074@sparrow.telecommunity.com>
	<4957f1ef1001090713l1ca97cd0k8254e9ed3ae4e34e@mail.gmail.com>
	<20100109163905.4671C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001091047i661d79c1tfeb379386d95989a@mail.gmail.com>
	<4957f1ef1001091051p6f700bb9i27c43b5143bcc907@mail.gmail.com>
	<87ljg784wc.fsf@wks.lan>
	<319e029f1001091148j3c6a929g6867a89efb5da776@mail.gmail.com>
Message-ID: <65e0bb521001091504m43dd8ccctd3e41231033971c0@mail.gmail.com>

On Sat, Jan 9, 2010 at 2:48 PM, Lennart Regebro <regebro at gmail.com> wrote:
> On Sat, Jan 9, 2010 at 20:17, Suno Ano <suno.ano at sunoano.org> wrote:
>
>> ?- *package* means a Python package, (directory intended to be on
>> ? sys.path, with an __init__.py. We *never* mean a distributable
>> ? or installable archive, except when ?impedance matching? with
>> ? folks who think in terms of operating system parcels.
>>
>> ?- *parcel* is such a distributable / installable archive:
>> ? either in source form (an ?sdist?), or one of the binary
>> ? forms (egg., etc.). Any parcel may contain multiple
>> ? packages (or even no packages, in the case of standalone
>> ? scripts).
>>
>> ?- *project* is the process / community which produces releases of
>> ? a given set of software, identified by a name unique within
>> ? PyPI?s namespace. PyPI manages metadata about projects (names,
>> ? owners) and their releases. Every real project has at least
>> ? one release.
>>
>> ?- *release* is a set of one or more parcels of a project,
>> ? each sharing the same version. Some PyPI metadata is specific
>> ? to a release, rather than a project. Every release has at
>> ? least one parcel.
>>
>> ?- *installer* is an OS specific piece of software provides by
>> ? some project which usually installs a Python interpreter and
>> ? other general software in order to have some Python
>> ? application installed from scratch.
>
> Yes. Again, exactly how I use the words already. This is the same for
> all intents and purposes as Tareks proposal, this is how the words are
> being used already in practice.

I would also add the common use of the term "distribution" to that
glossary as well.

At http://python.org/download/ , that big software archive that you
download, unpack, and install---giving you a Python installed on your
system---is referred to as a "distribution". Ex., "the python.org
Python distribution", "the ActiveState Python distribution", etc.

Enthought calls theirs a distribution as well:
http://enthought.com/products/epd.php

Incidentally, Perl calls theirs distributions too:
http://www.cpan.org/ (source code distribution available here
http://www.cpan.org/src/README.html ).

Ruby calls theirs distributions as well http://www.ruby-lang.org/en/downloads/ .

---John

From glyph at twistedmatrix.com  Fri Jan  8 07:50:38 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Fri, 8 Jan 2010 01:50:38 -0500
Subject: [Catalog-sig] [Distutils]   packaging terminology confusion
In-Reply-To: <65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
Message-ID: <723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>

On Jan 7, 2010, at 10:43 PM, John Gabriele wrote:

> 3. People don't like calling those MyProject-1.0.2.tgz thingies
> "distributions". They keep calling them packages, and when you correct
> them, they say, "[sigh], fine. [eyes roll] 'distribution'."

This is not my experience of the term.  It's more that most people *still* don't know what the term "distribution" means in this context, unless they've extensively read the distutils documentation.  Or that they're talking about Python "distributions" at the same time as what Linux distributions are doing with them, and are trying to avoid being ambiguous.

It sure would be nice if we could use a made-up word like "eggs" to refer to these things.  Too bad that's taken too :-\.

> So, here's a suggestion: just call both things (packages and
> distributions) "packages", but then agree to fully qualify the names
> when you need to avoid ambiguity, for example: "I just built a
> distribution-package (or "dist-package" for short) and included
> numerous module-packages in it."

Except, oops, "dist-package" *also* already has a conflicting jargon meaning as well.  This is what Ubuntu calls "packages distributed by the Linux distribution", i.e. Ubuntu or Debian.  See, for example, <https://lists.ubuntu.com/archives/bazaar/2009q1/054089.html>.  Given that they introduced the term specifically to disambiguate between the things Python already calls "distributions", it seems like introducing the same term to refer to the thing that they were originally talking about would have the opposite effect of what you intend.

Consulting a thesaurus yields one word that nobody has proposed yet: "Parcel".  Helpfully, it still starts with "P", so we could rename it the "Python Parcel Index".  I have a dim memory hearing this word in a jargon-y context before, but it's certainly considerably more obscure than "package", "distribution", "archive", et cetera.

Anybody else like that one?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100108/b0b9388c/attachment.htm>

From david.lyon at preisshare.net  Mon Jan 11 01:57:07 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 11 Jan 2010 11:57:07 +1100 (EST)
Subject: [Catalog-sig] [Distutils]   packaging terminology confusion
In-Reply-To: <723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
Message-ID: <3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>


Hi Glyph,

> It sure would be nice if we could use a made-up word like "eggs" to refer
> to these things.  Too bad that's taken too :-\.

Yes.

"eggs" is the best name anybody could hope for to describe a package. It
already has general acceptance to a large degree amongst users (despite
it's faults).

Even Tarek has adopted it by forking setuptools. Tarek has complete say on
the direction of the .EGG

What needs to be done is for a decision to occur to either kill or revive
the "egg" in 2010.

Let's either fix them, or kill them. Fixing the "egg" means that Tarek has
to "evolve" Distribute. Which is well within his power. Killing them means
taking the egg references out of existing peps and packages and pypi and
starting again. That's also possible.

As a regular developer, I'd call for a "L'Oeuf incredible". Excuse my bad
french. A new egg to replace all the bad old eggs.

We need more simplicity in packaging in python..

"eggs" are cool. They're simple. They're what users want. They're what
pypi needs.

I just don't know why it is that there is such a big desire in python-core
to chuck out the fun and simple and try to replace it with complicated and
confusing. It's so *not* monty python..

If the eggs are going to be thrown out... chuck those forkers out. But I
think it would be a big mistake.

David







From jmg3000 at gmail.com  Mon Jan 11 16:17:13 2010
From: jmg3000 at gmail.com (John Gabriele)
Date: Mon, 11 Jan 2010 10:17:13 -0500
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
In-Reply-To: <C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
	<3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
	<C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>
Message-ID: <65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>

On Mon, Jan 11, 2010 at 9:19 AM, Barry Warsaw <barry at python.org> wrote:
> On Jan 10, 2010, at 7:57 PM, David Lyon wrote:
>
>> As a regular developer, I'd call for a "L'Oeuf incredible". Excuse my bad
>> french. A new egg to replace all the bad old eggs.
>>
>> We need more simplicity in packaging in python..
>>
>> "eggs" are cool. They're simple. They're what users want. They're what
>> pypi needs.
>
> +1
>

Do you mean, change the general name of these packaged up things from
"distributions" to "eggs"? So, we'd generically refer to, say,
"CheesyComestible-1.2.3.tgz" as an egg?

Interesting.

What term would you use to refer specifically to a ".egg" file?

---John

From bradallen137 at gmail.com  Mon Jan 11 16:46:08 2010
From: bradallen137 at gmail.com (Brad Allen)
Date: Mon, 11 Jan 2010 09:46:08 -0600
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
In-Reply-To: <65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
	<3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
	<C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>
	<65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>
Message-ID: <4957f1ef1001110746u1a5017dq62b76690fbb7f4c4@mail.gmail.com>

On Mon, Jan 11, 2010 at 9:17 AM, John Gabriele <jmg3000 at gmail.com> wrote:

> Do you mean, change the general name of these packaged up things from
> "distributions" to "eggs"? So, we'd generically refer to, say,
> "CheesyComestible-1.2.3.tgz" as an egg?
>
> Interesting.
>
> What term would you use to refer specifically to a ".egg" file?

Boiled egg. :-)

From barry at python.org  Mon Jan 11 15:19:14 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 11 Jan 2010 09:19:14 -0500
Subject: [Catalog-sig] [Distutils]   packaging terminology confusion
In-Reply-To: <3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
	<3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
Message-ID: <C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>

On Jan 10, 2010, at 7:57 PM, David Lyon wrote:

> As a regular developer, I'd call for a "L'Oeuf incredible". Excuse my bad
> french. A new egg to replace all the bad old eggs.
> 
> We need more simplicity in packaging in python..
> 
> "eggs" are cool. They're simple. They're what users want. They're what
> pypi needs.

+1

-Barry


From zooko at zooko.com  Mon Jan 11 18:08:43 2010
From: zooko at zooko.com (Zooko Wilcox-O'Hearn)
Date: Mon, 11 Jan 2010 10:08:43 -0700
Subject: [Catalog-sig] [Distutils]   packaging terminology confusion
In-Reply-To: <65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<20100107164433.662DE3A4074@sparrow.telecommunity.com>
	<4B464207.3030507@v.loewis.de>
	<20100107204102.D121C3A4074@sparrow.telecommunity.com>
	<4957f1ef1001071329m2de42256k31581cf2c6d0a1e0@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
Message-ID: <06BFB135-97FE-4324-9899-6388C03CD303@zooko.com>

On Thursday, 2010-01-07, at 20:43 , John Gabriele wrote:

> So, here's a suggestion: just call both things (packages and  
> distributions) "packages", but then agree to fully qualify the  
> names when you need to avoid ambiguity, for example: "I just built  
> a distribution-package (or "dist-package" for short) and included  
> numerous module-packages in it."

+1

Thank you.

Your proposal has the advantage that it is what most users are  
already doing and will continue to do whether we like it or not.  :-)

> 3. People don't like calling those MyProject-1.0.2.tgz thingies  
> "distributions". They keep calling them packages, and when you  
> correct them, they say, "[sigh], fine. [eyes roll] 'distribution'."

Indeed, for most of the people I talk to (most of whom are not core  
committers to the Python project), I say "You want such-and-such a  
tool?  Please download the such-and-such package. See http:// 
pypi.python.org .", and they say "Ok".  If I say "Please download the  
such-and-such distribution." they say "Huh?".

This is a point that perhaps needs to be repeated.  We as distutils- 
sig don't have the option of educating all Python users about our  
peculiar terminology.  Maybe ten years ago, but today there are too  
many of them.  We have the option of either carrying on status quo  
with recurring confusion and clarification, or adopting the  
terminology that people already expect.

Regards,

Zooko

From chris at simplistix.co.uk  Mon Jan 11 20:49:54 2010
From: chris at simplistix.co.uk (Chris Withers)
Date: Mon, 11 Jan 2010 19:49:54 +0000
Subject: [Catalog-sig] PyPI down?
Message-ID: <4B4B80E2.60804@simplistix.co.uk>

Hi All,

Anyone know whassup with PyPI?
Slow to the point of timing out for me...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From martin at v.loewis.de  Mon Jan 11 21:38:53 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 11 Jan 2010 21:38:53 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4B80E2.60804@simplistix.co.uk>
References: <4B4B80E2.60804@simplistix.co.uk>
Message-ID: <4B4B8C5D.3040409@v.loewis.de>

> Anyone know whassup with PyPI?

Works fine for me.

Regards,
Martin

From chris at simplistix.co.uk  Mon Jan 11 21:41:35 2010
From: chris at simplistix.co.uk (Chris Withers)
Date: Mon, 11 Jan 2010 20:41:35 +0000
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4B8C5D.3040409@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
Message-ID: <4B4B8CFF.9050505@simplistix.co.uk>

Martin v. L?wis wrote:
>> Anyone know whassup with PyPI?
> 
> Works fine for me.

I'm still seeing really variable performance and only from PyPI. The 
main python website and all other websites I've tried are fine.

Are there any site load stats easily accessible? I wonder if someone is 
maybe spidering PyPI in an antisocial way or some such...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From ziade.tarek at gmail.com  Mon Jan 11 22:07:46 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 11 Jan 2010 22:07:46 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4B8CFF.9050505@simplistix.co.uk>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
Message-ID: <94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>

On Mon, Jan 11, 2010 at 9:41 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> Martin v. L?wis wrote:
>>>
>>> Anyone know whassup with PyPI?
>>
>> Works fine for me.
>
> I'm still seeing really variable performance and only from PyPI. The main
> python website and all other websites I've tried are fine.
>
> Are there any site load stats easily accessible? I wonder if someone is
> maybe spidering PyPI in an antisocial way or some such...

Have you tried a traceroute ? Maybe a node on your way is somehow very
slow today.

>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
> ? ? ? ? ? - http://www.simplistix.co.uk
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>



-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Mon Jan 11 22:09:34 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 11 Jan 2010 22:09:34 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4B8CFF.9050505@simplistix.co.uk>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
Message-ID: <4B4B938E.1040903@v.loewis.de>

> Are there any site load stats easily accessible?

http://pypi.python.org/munin/localdomain/localhost.localdomain.html

Martin

From chris at simplistix.co.uk  Mon Jan 11 22:12:18 2010
From: chris at simplistix.co.uk (Chris Withers)
Date: Mon, 11 Jan 2010 21:12:18 +0000
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>	
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
Message-ID: <4B4B9432.9050704@simplistix.co.uk>

Tarek Ziad? wrote:
>> I'm still seeing really variable performance and only from PyPI. The main
>> python website and all other websites I've tried are fine.
>>
>> Are there any site load stats easily accessible? I wonder if someone is
>> maybe spidering PyPI in an antisocial way or some such...
> 
> Have you tried a traceroute ? Maybe a node on your way is somehow very
> slow today.

Yup, nothing out of the ordinary... oh well, I guess it is just me and 
I'll wait and hope things get better ;-)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From ziade.tarek at gmail.com  Mon Jan 11 22:22:23 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 11 Jan 2010 22:22:23 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4B9432.9050704@simplistix.co.uk>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
Message-ID: <94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>

2010/1/11 Chris Withers <chris at simplistix.co.uk>:
> Tarek Ziad? wrote:
>>>
>>> I'm still seeing really variable performance and only from PyPI. The main
>>> python website and all other websites I've tried are fine.
>>>
>>> Are there any site load stats easily accessible? I wonder if someone is
>>> maybe spidering PyPI in an antisocial way or some such...
>>
>> Have you tried a traceroute ? Maybe a node on your way is somehow very
>> slow today.
>
> Yup, nothing out of the ordinary... oh well, I guess it is just me and I'll
> wait and hope things get better ;-)

If we have time at Pycon, and if some folks are interested, I would
like to finish the implementation of PEP 381,

and if some folks in the Pip team are interested, see if we can
develop a clisent-side geolocalisation
thingie to pick the closest PyPI mirror.

Cheers
Tarek

-- 
Tarek Ziad? | http://ziade.org

From tjreedy at udel.edu  Mon Jan 11 22:45:23 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Jan 2010 16:45:23 -0500
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4B8CFF.9050505@simplistix.co.uk>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
Message-ID: <hig65h$d58$1@ger.gmane.org>

On 1/11/2010 3:41 PM, Chris Withers wrote:
> Martin v. L?wis wrote:
>>> Anyone know whassup with PyPI?
>>
>> Works fine for me.
>
> I'm still seeing really variable performance and only from PyPI. The
> main python website and all other websites I've tried are fine.
>
> Are there any site load stats easily accessible? I wonder if someone is
> maybe spidering PyPI in an antisocial way or some such...

Currently, (4:18 est, Delaware, USA) I have gotten several pages with 1 
sec response.



From tjreedy at udel.edu  Mon Jan 11 22:50:02 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Jan 2010 16:50:02 -0500
Subject: [Catalog-sig] Commercial software listings?
Message-ID: <hig6e7$e5m$1@ger.gmane.org>

Is PyPI intended for the listing of paid commercial software?

Given "If you have some free software or open source modules that you've 
polished up and would like to contribute, we'd love to have them 
included in the PyPI!" from

http://wiki.python.org/moin/CheeseShopTutorial

I thought not, but

http://pypi.python.org/pypi/emma/1.0

for instance, has a 1000 euro license fee. If this is OK, I think there 
should be an indication on such listings so one not interested in paying 
such a fee would not waste time. Or there should be a separate 
'commercial software' section with paid listings.

tjr


From martin at v.loewis.de  Mon Jan 11 23:06:20 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 11 Jan 2010 23:06:20 +0100
Subject: [Catalog-sig] PyPI ssh access
Message-ID: <4B4BA0DC.5060209@v.loewis.de>

I have now set up SSH access for PyPI. The procedure works like this:

1. upload your SSH key(s) to your PyPI account.
2. Connect using 'ssh -T submit at pypi.python.org', and send a single HTTP
   request. PyPI will associate the request with the respective PyPI
   account.

I have yet to figure out why Connection: keep-alive doesn't work (i.e.
you need to reconnect for every request).

Apart from that, please try this out and report your findings; I'm
sure patches to distutils/distribute are welcome.

If you find bugs, please report them to the PyPI bugtracker.

Regards,
Martin

From martin at v.loewis.de  Mon Jan 11 23:30:35 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 11 Jan 2010 23:30:35 +0100
Subject: [Catalog-sig] Commercial software listings?
In-Reply-To: <hig6e7$e5m$1@ger.gmane.org>
References: <hig6e7$e5m$1@ger.gmane.org>
Message-ID: <4B4BA68B.3090607@v.loewis.de>

> Is PyPI intended for the listing of paid commercial software?

I see nothing wrong with that.

> If this is OK, I think there
> should be an indication on such listings so one not interested in paying
> such a fee would not waste time. Or there should be a separate
> 'commercial software' section with paid listings.

Who would police such a requirement?

Regards,
Martin

From robert.kern at gmail.com  Mon Jan 11 23:48:37 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 11 Jan 2010 16:48:37 -0600
Subject: [Catalog-sig] Commercial software listings?
In-Reply-To: <hig6e7$e5m$1@ger.gmane.org>
References: <hig6e7$e5m$1@ger.gmane.org>
Message-ID: <hig9s4$pn9$1@ger.gmane.org>

On 2010-01-11 15:50 PM, Terry Reedy wrote:
> Is PyPI intended for the listing of paid commercial software?
>
> Given "If you have some free software or open source modules that you've
> polished up and would like to contribute, we'd love to have them
> included in the PyPI!" from
>
> http://wiki.python.org/moin/CheeseShopTutorial
>
> I thought not, but
>
> http://pypi.python.org/pypi/emma/1.0
>
> for instance, has a 1000 euro license fee. If this is OK, I think there
> should be an indication on such listings so one not interested in paying
> such a fee would not waste time. Or there should be a separate
> 'commercial software' section with paid listings.

Most of these packages, including emma, correctly have the "License :: 
Other/Proprietary License" classifier attached to them.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ssteinerx at gmail.com  Mon Jan 11 23:56:48 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 11 Jan 2010 17:56:48 -0500
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4B8CFF.9050505@simplistix.co.uk>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
Message-ID: <6B2F737B-29C5-4EF0-8AC6-FE993D892635@gmail.com>


On Jan 11, 2010, at 3:41 PM, Chris Withers wrote:

> Martin v. L?wis wrote:
>>> Anyone know whassup with PyPI?
>> Works fine for me.
> 
> I'm still seeing really variable performance and only from PyPI. The main python website and all other websites I've tried are fine.
> 
> Are there any site load stats easily accessible? I wonder if someone is maybe spidering PyPI in an antisocial way or some such...

I was spidering PyPI antisocially yesterday trying to get a clean initial capture for the PyPI metadata mirror but I swear I've not touched it today!

Maybe there's a memory leak that has carry-over effects?

S


From ben+python at benfinney.id.au  Tue Jan 12 00:04:59 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 12 Jan 2010 10:04:59 +1100
Subject: [Catalog-sig] Commercial software listings?
References: <hig6e7$e5m$1@ger.gmane.org>
Message-ID: <87k4vos0o4.fsf@benfinney.id.au>

Terry Reedy <tjreedy at udel.edu> writes:

> Is PyPI intended for the listing of paid commercial software?

I would think that's fine, so long as the paid commercial software is
free software. The price paid by the recipient is orthogonal to the
freedom of the recipient in the work.

Perhaps you want instead to talk about software works that are not free;
the usual term for this is ?non-free? or ?proprietary? software.

I would very much prefer if PyPI were an index of free software works
for Python, and that all such works were available for download from the
site under free terms.

> I thought not, but
>
> http://pypi.python.org/pypi/emma/1.0
>
> for instance, has a 1000 euro license fee.

Regardless of the fee (I've happily paid fees for free software), the
license terms for that work appear to be non-free.

It seems, therefore, that PyPI also promotes non-free software. I would
prefer otherwise, but I don't get a vote.

> If this is OK, I think there should be an indication on such listings
> so one not interested in paying such a fee would not waste time. Or
> there should be a separate 'commercial software' section with paid
> listings.

In numerous cases, one party can distribute non-free software for a fee,
a different party can distribute the same non-free software for a much
different fee, and yet another party can distribute the same non-free
software for no fee. Again, the fee charged is orthogonal to the work's
freeness or otherwise.

I don't know if any such works actually exist yet on PyPI, but it's
clearly feasible. Given that such a work would have only one entry on
the PyPI, how (and why) would you have PyPI differentiate based on
whether some party, somewhere, charges a fee?

The license terms already have a field (?License?), and the trove
classification indicates whether the license terms are non-free
(?License :: Other/Proprietary License? in the case of the work you're
referring to). That seems enough to make the decision.

-- 
 \                   ?????????? (What is undesirable to you, |
  `\                                             do not do to others.) |
_o__)                             ???? Confucius, 551 BCE ? 479 BCE |
Ben Finney


From ssteinerx at gmail.com  Tue Jan 12 00:08:28 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 11 Jan 2010 18:08:28 -0500
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
Message-ID: <5962D3B9-12CB-4922-9E4C-02841561BF74@gmail.com>


On Jan 11, 2010, at 4:22 PM, Tarek Ziad? wrote:
> If we have time at Pycon, and if some folks are interested, I would
> like to finish the implementation of PEP 381,

Tarek,

	I'm working on mirroring the meta-data as per a conversation last week on python-dev (I think).

	Right now I'm just putting up a zipped copy of the metadata pickle but it is be generic enough, and will include a library for pulling down a copy of the meta data from a mirror and updating it by date.

	It's a partial solution to the sync issue and maybe could be used as the first part of the "sync meta-data" in the mirroring protocol.

S


From ben+python at benfinney.id.au  Tue Jan 12 00:06:40 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 12 Jan 2010 10:06:40 +1100
Subject: [Catalog-sig] PyPI ssh access
References: <4B4BA0DC.5060209@v.loewis.de>
Message-ID: <87fx6cs0lb.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> The procedure works like this:

I can only assume that this message is the primary place where the
procedure is documented; if there is a more thorough URL describing it,
that would be good to know.

> 1. upload your SSH key(s) to your PyPI account.

How?

-- 
 \       ?I love and treasure individuals as I meet them, I loathe and |
  `\     despise the groups they identify with and belong to.? ?George |
_o__)                                                     Carlin, 2007 |
Ben Finney


From martin at v.loewis.de  Tue Jan 12 00:17:36 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 00:17:36 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <6B2F737B-29C5-4EF0-8AC6-FE993D892635@gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<6B2F737B-29C5-4EF0-8AC6-FE993D892635@gmail.com>
Message-ID: <4B4BB190.8090106@v.loewis.de>

> Maybe there's a memory leak that has carry-over effects?

Unlikely - it will restart at least once a day, but likely more often
(i.e. every 1000 requests, which is fairly often).

Regards,
Martin

From martin at v.loewis.de  Tue Jan 12 00:23:06 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 12 Jan 2010 00:23:06 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <87fx6cs0lb.fsf@benfinney.id.au>
References: <4B4BA0DC.5060209@v.loewis.de> <87fx6cs0lb.fsf@benfinney.id.au>
Message-ID: <4B4BB2DA.9040305@v.loewis.de>

>> The procedure works like this:
> 
> I can only assume that this message is the primary place where the
> procedure is documented; if there is a more thorough URL describing it,
> that would be good to know.

No. I don't expect end users to actually use it - primary "users" would
be authors of software that automatically interacts with PyPI.

Feel free to put it into the Wiki, though.

>> 1. upload your SSH key(s) to your PyPI account.
> 
> How?

I would *really* like you to guess here. Seriously. If you guess wrong,
let me know what your guess was, and I see whether I can arrange to
make it work.

Having it intuitively right is much better than having people read
documentation (where they then have to guess how they can get to
the documentation, or that the documentation exists in the first
place).

Regards,
Martin


From exarkun at twistedmatrix.com  Tue Jan 12 00:30:49 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Mon, 11 Jan 2010 23:30:49 -0000
Subject: [Catalog-sig] Error trying to log in with OpenID
Message-ID: <20100111233049.1898.874891027.divmod.xquotient.41@localhost.localdomain>

I got this page just now trying to log in to PyPI (from 
<http://pypi.python.org/pypi?:action=openid>):

Error...

There's been a problem with your request

: local variable 'services' referenced before assignment

Jean-Paul

From ben+python at benfinney.id.au  Tue Jan 12 00:43:20 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 12 Jan 2010 10:43:20 +1100
Subject: [Catalog-sig] PyPI ssh access
References: <4B4BA0DC.5060209@v.loewis.de> <87fx6cs0lb.fsf@benfinney.id.au>
	<4B4BB2DA.9040305@v.loewis.de>
Message-ID: <877hroryw7.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> >> 1. upload your SSH key(s) to your PyPI account.
> > 
> > How?
>
> I would *really* like you to guess here. Seriously. If you guess
> wrong, let me know what your guess was, and I see whether I can
> arrange to make it work.

By analogy with how SSH works on other systems, I would expect to use
some other file upload protocol (e.g. FTP) to place a file named
?authorized_keys? in my home directory, containing the contents of my
SSH public key.

But most of that doesn't apply here: I don't have an alternate arbitrary
file-upload protocol to PyPI, only an ?upload a new package version?
tool, which doesn't fit. I also don't know of a ?home directory? for my
account.

-- 
 \           ?In case you haven't noticed, [the USA] are now almost as |
  `\     feared and hated all over the world as the Nazis were.? ?Kurt |
_o__)                                                   Vonnegut, 2004 |
Ben Finney


From jeremy.kloth at gmail.com  Tue Jan 12 00:59:09 2010
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Mon, 11 Jan 2010 16:59:09 -0700
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <87fx6cs0lb.fsf@benfinney.id.au>
References: <4B4BA0DC.5060209@v.loewis.de> <87fx6cs0lb.fsf@benfinney.id.au>
Message-ID: <201001111659.09296.jeremy.kloth@gmail.com>

On Monday 11 January 2010 04:06:40 pm Ben Finney wrote:
> "Martin v. L?wis" <martin at v.loewis.de> writes:
> > 1. upload your SSH key(s) to your PyPI account.
> 
> How?

My first guess would be to upload via a submission box on my account page, 
just like how SourceForge had SSH access implemented.

And for the record, this is how PyPI has it implemented.

-- 
Jeremy Kloth
http://4suite.org/

From david.lyon at preisshare.net  Tue Jan 12 01:09:53 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 12 Jan 2010 11:09:53 +1100 (EST)
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
In-Reply-To: <65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
	<3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
	<C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>
	<65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>
Message-ID: <1289.218.214.45.58.1263254993.squirrel@syd-srv02.ezyreg.com>

> On Mon, Jan 11, 2010 at 9:19 AM, Barry Warsaw <barry at python.org> wrote:

> Do you mean, change the general name of these packaged up things from
> "distributions" to "eggs"?

What I mean is that the egg concept abstracts all the packaging
details from the user extremely well.

If a user gets told that all python packages come as python
eggs, they just accept that and move on.

If we could move to a position (slowly) whereby we had all
packages as .egg files, that would make life simpler for
users. It is less choices.

> So, we'd generically refer to, say,
> "CheesyComestible-1.2.3.tgz" as an egg?

No.

Just "CheesyComestible-1.2.3.egg"

Eliminate : CheesyComestible-1.2.3.zip CheesyComestible-1.2.3.exe
CheesyComestible-1.2.3.tar.gz CheesyComestible-1.2.3.bz2

Unneccessary and confusing.

> What term would you use to refer specifically to a ".egg" file?

Just "Python Egg".

David





From ssteinerx at gmail.com  Tue Jan 12 01:25:33 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 11 Jan 2010 19:25:33 -0500
Subject: [Catalog-sig] Error trying to log in with OpenID
In-Reply-To: <20100111233049.1898.874891027.divmod.xquotient.41@localhost.localdomain>
References: <20100111233049.1898.874891027.divmod.xquotient.41@localhost.localdomain>
Message-ID: <C8FC05C4-138B-4D1E-B250-2C4453EDD0E2@gmail.com>


On Jan 11, 2010, at 6:30 PM, exarkun at twistedmatrix.com wrote:

> I got this page just now trying to log in to PyPI (from <http://pypi.python.org/pypi?:action=openid>):
> 
> Error...
> 
> There's been a problem with your request
> 
> : local variable 'services' referenced before assignment

I get over to myopenid, then get this:

Error...

There's been a problem with your request

: 'NoneType' object has no attribute 'split'

S


From doug.hellmann at gmail.com  Tue Jan 12 01:20:35 2010
From: doug.hellmann at gmail.com (Doug Hellmann)
Date: Mon, 11 Jan 2010 19:20:35 -0500
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <4B4BA0DC.5060209@v.loewis.de>
References: <4B4BA0DC.5060209@v.loewis.de>
Message-ID: <9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>


On Jan 11, 2010, at 5:06 PM, Martin v. L?wis wrote:

> I have now set up SSH access for PyPI. The procedure works like this:

Sorry if this should be obvious, but what does ssh access to PyPI give  
me, as a developer?  Can I manage uploads that way or something?

Thanks,
Doug


From ben+python at benfinney.id.au  Tue Jan 12 02:33:25 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 12 Jan 2010 12:33:25 +1100
Subject: [Catalog-sig] PyPI ssh access
References: <4B4BA0DC.5060209@v.loewis.de> <87fx6cs0lb.fsf@benfinney.id.au>
	<201001111659.09296.jeremy.kloth@gmail.com>
Message-ID: <87skacqf8a.fsf@benfinney.id.au>

Jeremy Kloth <jeremy.kloth at gmail.com> writes:

> On Monday 11 January 2010 04:06:40 pm Ben Finney wrote:
> > "Martin v. L?wis" <martin at v.loewis.de> writes:
> > > 1. upload your SSH key(s) to your PyPI account.
> > 
> > How?
>
> My first guess would be to upload via a submission box on my account
> page,

I almost never visit my PyPI account page, since that's not my primary
interaction with PyPI as a package maintainer. (The Distutils library is
my primary interface with PyPI as a package maintainer.) So that's not
something I'd guess.

> just like how SourceForge had SSH access implemented.

I haven't used SourceForge for over a decade (and their login form never
appears for me anyway), so this analogy would not occur to me.

-- 
 \      ?When I wake up in the morning, I just can't get started until |
  `\     I've had that first, piping hot pot of coffee. Oh, I've tried |
_o__)                                    other enemas...? ?Emo Philips |
Ben Finney


From ben+python at benfinney.id.au  Tue Jan 12 02:38:23 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 12 Jan 2010 12:38:23 +1100
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
	<3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
	<C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>
	<65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>
	<1289.218.214.45.58.1263254993.squirrel@syd-srv02.ezyreg.com>
Message-ID: <87k4voqf00.fsf@benfinney.id.au>

"David Lyon" <david.lyon at preisshare.net> writes:

> Just "CheesyComestible-1.2.3.egg"
>
> Eliminate : CheesyComestible-1.2.3.zip CheesyComestible-1.2.3.exe
> CheesyComestible-1.2.3.tar.gz CheesyComestible-1.2.3.bz2
>
> Unneccessary and confusing.

How are they unnecessary? There needs to be, at least, a difference
between the source package and the binary package. Further, you (IIRC)
have been arguing for Windows executable installers, which are
necessarily going to be different from either the source package or the
binary package for non-Windows systems.

All of them *are* ?the package?, at a particular version, in different
and necessary forms.

-- 
 \      ?It is the integrity of each individual human that is in final |
  `\        examination. On personal integrity hangs humanity's fate.? |
_o__)               ?Richard Buckminster Fuller, _Critical Path_, 1981 |
Ben Finney


From ben+python at benfinney.id.au  Tue Jan 12 02:34:47 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 12 Jan 2010 12:34:47 +1100
Subject: [Catalog-sig] PyPI ssh access
References: <4B4BA0DC.5060209@v.loewis.de>
	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>
Message-ID: <87ocl0qf60.fsf@benfinney.id.au>

Doug Hellmann <doug.hellmann at gmail.com> writes:

> On Jan 11, 2010, at 5:06 PM, Martin v. L?wis wrote:
>
> > I have now set up SSH access for PyPI. The procedure works like
> > this:
>
> Sorry if this should be obvious, but what does ssh access to PyPI give
> me, as a developer? Can I manage uploads that way or something?

I'm rather confused too.

Would it not have been better to ask how people expect such a feature to
work, *before* implementing it?

-- 
 \      ?Saying that Java is nice because it works on all OSes is like |
  `\     saying that anal sex is nice because it works on all genders? |
_o__)                                                ?http://bash.org/ |
Ben Finney


From david.lyon at preisshare.net  Tue Jan 12 02:44:49 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 12 Jan 2010 12:44:49 +1100 (EST)
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
In-Reply-To: <87k4voqf00.fsf@benfinney.id.au>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
	<3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
	<C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>
	<65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>
	<1289.218.214.45.58.1263254993.squirrel@syd-srv02.ezyreg.com>
	<87k4voqf00.fsf@benfinney.id.au>
Message-ID: <1640.218.214.45.58.1263260689.squirrel@syd-srv02.ezyreg.com>

Ben Finney writes:
>> Eliminate : CheesyComestible-1.2.3.zip CheesyComestible-1.2.3.exe
>> CheesyComestible-1.2.3.tar.gz CheesyComestible-1.2.3.bz2
>>
>> Unneccessary and confusing.
>
> How are they unnecessary? There needs to be, at least, a difference
> between the source package and the binary package.

Why?

If you have a zip/archive file, you can put anything in it. No reason why
'everything' can't go in it.

A "L'Oeuf incredible" might include a Python 2.x and Python 3.x code set,
make code for linux, .pyd for windows.

It would be so un-confusing to have an egg like that.

> Further, you (IIRC)
> have been arguing for Windows executable installers, which are
> necessarily going to be different from either the source package or the
> binary package for non-Windows systems.

I don't like those. I'd prefer as a security issue thing to have a .egg
package, associated with python, that I can click on with my browser, and
download and install into python automatically.

> All of them *are* the package, at a particular version, in different
> and necessary forms.

It's just confusing that way. But I understand all the history.

We had divergence of all these package forms..

Now we need convergence to a new singular package form.

David





From ssteinerx at gmail.com  Tue Jan 12 04:05:44 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 11 Jan 2010 22:05:44 -0500
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
Message-ID: <FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>


On Jan 11, 2010, at 4:22 PM, Tarek Ziad? wrote:

> If we have time at Pycon, and if some folks are interested, I would
> like to finish the implementation of PEP 381,

Is z3c.pypimirror the "sample implementation" at this point?

Thanks,

S


From sridharr at activestate.com  Tue Jan 12 05:37:29 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Mon, 11 Jan 2010 20:37:29 -0800
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <hig65h$d58$1@ger.gmane.org>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B8C5D.3040409@v.loewis.de>	<4B4B8CFF.9050505@simplistix.co.uk>
	<hig65h$d58$1@ger.gmane.org>
Message-ID: <4B4BFC89.6060701@activestate.com>

On 1/11/2010 1:45 PM, Terry Reedy wrote:
> On 1/11/2010 3:41 PM, Chris Withers wrote:
>> Martin v. L?wis wrote:
>>>> Anyone know whassup with PyPI?
>>>
>>> Works fine for me.
>>
>> I'm still seeing really variable performance and only from PyPI. The
>> main python website and all other websites I've tried are fine.
>>
>> Are there any site load stats easily accessible? I wonder if someone is
>> maybe spidering PyPI in an antisocial way or some such...
>
> Currently, (4:18 est, Delaware, USA) I have gotten several pages with 1
> sec response.

The PyPM mirror program[1] runs at 0100hrs PST (= 0400hrs EST). What it 
does is to redownload the latest version of packages released in last 
two days (as reported by the `changelog` XMLRPC API). The download part 
is equivalent to running "easy_install $pkgname" sequentially - except 
duplicates are handled - for each of those packages and their 
dependencies. Only one XMLRPC API call is made per day.

I believe z3c.pypimirror - and soon PEP 381 - works in a similar fashion 
(except for less sophisticated scrapping logic).

It is not clear to me how this could overload the server. Does this 
happen everyday at the same time interval?

-srid

****
[1] apologies for not setting the user-agent string properly; I will get 
around to doing it tomorrow

From martin at v.loewis.de  Tue Jan 12 07:08:03 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 12 Jan 2010 07:08:03 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <201001111659.09296.jeremy.kloth@gmail.com>
References: <4B4BA0DC.5060209@v.loewis.de> <87fx6cs0lb.fsf@benfinney.id.au>
	<201001111659.09296.jeremy.kloth@gmail.com>
Message-ID: <4B4C11C3.3060405@v.loewis.de>

> My first guess would be to upload via a submission box on my account page, 
> just like how SourceForge had SSH access implemented.
> 
> And for the record, this is how PyPI has it implemented.

Apparently, you need to have used web upload of SSH keys elsewhere to
guess that this might be an option. Thanks for the confirmation that
people would then be able to find it.

Regards,
Martin


From martin at v.loewis.de  Tue Jan 12 07:11:24 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 07:11:24 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>
References: <4B4BA0DC.5060209@v.loewis.de>
	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>
Message-ID: <4B4C128C.6070004@v.loewis.de>

>> I have now set up SSH access for PyPI. The procedure works like this:
> 
> Sorry if this should be obvious, but what does ssh access to PyPI give
> me, as a developer?  Can I manage uploads that way or something?

In principle, you would be able to use all PyPI-related distutils
commands (register, upload) over SSH, which then would free you from
the need to provide a password (interactively, or on disk).

In practice, this would require distutils to be modified to actually
use that protocol, instead of using http over plain TCP.

Regards,
Martin

From martin at v.loewis.de  Tue Jan 12 07:05:00 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 12 Jan 2010 07:05:00 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <877hroryw7.fsf@benfinney.id.au>
References: <4B4BA0DC.5060209@v.loewis.de>
	<87fx6cs0lb.fsf@benfinney.id.au>	<4B4BB2DA.9040305@v.loewis.de>
	<877hroryw7.fsf@benfinney.id.au>
Message-ID: <4B4C110C.8000009@v.loewis.de>

>> I would *really* like you to guess here. Seriously. If you guess
>> wrong, let me know what your guess was, and I see whether I can
>> arrange to make it work.
> 
> By analogy with how SSH works on other systems, I would expect to use
> some other file upload protocol (e.g. FTP) to place a file named
> ?authorized_keys? in my home directory, containing the contents of my
> SSH public key.
> 
> But most of that doesn't apply here: I don't have an alternate arbitrary
> file-upload protocol to PyPI, only an ?upload a new package version?
> tool, which doesn't fit. I also don't know of a ?home directory? for my
> account.

I see. That's sad. Please have a look at the "Your details" link in PyPI,
http://pypi.python.org/pypi?%3Aaction=user_form

Regards,
Martin

From martin at v.loewis.de  Tue Jan 12 07:16:23 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 12 Jan 2010 07:16:23 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <87ocl0qf60.fsf@benfinney.id.au>
References: <4B4BA0DC.5060209@v.loewis.de>	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>
	<87ocl0qf60.fsf@benfinney.id.au>
Message-ID: <4B4C13B7.4040605@v.loewis.de>

> I'm rather confused too.
> 
> Would it not have been better to ask how people expect such a feature to
> work, *before* implementing it?

But we did discuss it, on this list, many times. I only implemented it
because people had been asking for it.

Regards,
Martin


From martin at v.loewis.de  Tue Jan 12 07:17:28 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 07:17:28 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>
Message-ID: <4B4C13F8.5090707@v.loewis.de>

>> If we have time at Pycon, and if some folks are interested, I would
>> like to finish the implementation of PEP 381,
> 
> Is z3c.pypimirror the "sample implementation" at this point?

No, it's not - although it's fairly close.

Regards,
Martin

From doug.hellmann at gmail.com  Tue Jan 12 14:11:42 2010
From: doug.hellmann at gmail.com (Doug Hellmann)
Date: Tue, 12 Jan 2010 08:11:42 -0500
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <4B4C128C.6070004@v.loewis.de>
References: <4B4BA0DC.5060209@v.loewis.de>
	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>
	<4B4C128C.6070004@v.loewis.de>
Message-ID: <F08ABF9E-FCD4-4D1B-BAAC-4CF70787D471@gmail.com>


On Jan 12, 2010, at 1:11 AM, Martin v. L?wis wrote:

>>> I have now set up SSH access for PyPI. The procedure works like  
>>> this:
>>
>> Sorry if this should be obvious, but what does ssh access to PyPI  
>> give
>> me, as a developer?  Can I manage uploads that way or something?
>
> In principle, you would be able to use all PyPI-related distutils
> commands (register, upload) over SSH, which then would free you from
> the need to provide a password (interactively, or on disk).

Ah, nice!

> In practice, this would require distutils to be modified to actually
> use that protocol, instead of using http over plain TCP.

Are those changes planned?

Doug


From mal at egenix.com  Tue Jan 12 14:25:35 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 12 Jan 2010 14:25:35 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <4B4BA0DC.5060209@v.loewis.de>
References: <4B4BA0DC.5060209@v.loewis.de>
Message-ID: <4B4C784F.8000705@egenix.com>

"Martin v. L?wis" wrote:
> I have now set up SSH access for PyPI.

Could you please summarize why SSH access to PyPI may be a
useful feature to have and what the intended use is ?

> The procedure works like this:
> 
> 1. upload your SSH key(s) to your PyPI account.
> 2. Connect using 'ssh -T submit at pypi.python.org', and send a single HTTP
>    request. PyPI will associate the request with the respective PyPI
>    account.
> 
> I have yet to figure out why Connection: keep-alive doesn't work (i.e.
> you need to reconnect for every request).
> 
> Apart from that, please try this out and report your findings; I'm
> sure patches to distutils/distribute are welcome.
> 
> If you find bugs, please report them to the PyPI bugtracker.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 12 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ssteinerx at gmail.com  Tue Jan 12 14:34:52 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 12 Jan 2010 08:34:52 -0500
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4C13F8.5090707@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>
	<4B4C13F8.5090707@v.loewis.de>
Message-ID: <AC944FA0-9EC3-4E56-B4C9-44FE68FCAE69@gmail.com>


On Jan 12, 2010, at 1:17 AM, Martin v. L?wis wrote:

>>> If we have time at Pycon, and if some folks are interested, I would
>>> like to finish the implementation of PEP 381,
>> 
>> Is z3c.pypimirror the "sample implementation" at this point?
> 
> No, it's not - although it's fairly close.

I've been looking at it to possibly use as part of the PyPI meta-data mirroring tool we discussed just before new years.

Since mirroring the meta-data would be an efficient first step towards any actual data mirroring, should I use the z3c.pypimirror code and improve that (better granularity, database caching with sqlite instead of .pkl, etc.) or start afresh.

Is it being considered as the starting point, or not?

S


From ziade.tarek at gmail.com  Tue Jan 12 14:39:59 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 12 Jan 2010 14:39:59 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <F08ABF9E-FCD4-4D1B-BAAC-4CF70787D471@gmail.com>
References: <4B4BA0DC.5060209@v.loewis.de>
	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>
	<4B4C128C.6070004@v.loewis.de>
	<F08ABF9E-FCD4-4D1B-BAAC-4CF70787D471@gmail.com>
Message-ID: <94bdd2611001120539gbcb4965q9701c5ec906191de@mail.gmail.com>

On Tue, Jan 12, 2010 at 2:11 PM, Doug Hellmann <doug.hellmann at gmail.com> wrote:
>
> On Jan 12, 2010, at 1:11 AM, Martin v. L?wis wrote:
>
>>>> I have now set up SSH access for PyPI. The procedure works like this:
>>>
>>> Sorry if this should be obvious, but what does ssh access to PyPI give
>>> me, as a developer? ?Can I manage uploads that way or something?
>>
>> In principle, you would be able to use all PyPI-related distutils
>> commands (register, upload) over SSH, which then would free you from
>> the need to provide a password (interactively, or on disk).
>
> Ah, nice!
>
>> In practice, this would require distutils to be modified to actually
>> use that protocol, instead of using http over plain TCP.
>
> Are those changes planned?

I can add this to Distutils' upload command (and Distribute until
Distutils is released standalone)
as soon as people have tried it manually successfully.

Tarek

-- 
Tarek Ziad? | http://ziade.org

From mal at egenix.com  Tue Jan 12 14:56:45 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 12 Jan 2010 14:56:45 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <94bdd2611001120539gbcb4965q9701c5ec906191de@mail.gmail.com>
References: <4B4BA0DC.5060209@v.loewis.de>	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>	<4B4C128C.6070004@v.loewis.de>	<F08ABF9E-FCD4-4D1B-BAAC-4CF70787D471@gmail.com>
	<94bdd2611001120539gbcb4965q9701c5ec906191de@mail.gmail.com>
Message-ID: <4B4C7F9D.5070703@egenix.com>

Tarek Ziad? wrote:
> On Tue, Jan 12, 2010 at 2:11 PM, Doug Hellmann <doug.hellmann at gmail.com> wrote:
>>
>> On Jan 12, 2010, at 1:11 AM, Martin v. L?wis wrote:
>>
>>>>> I have now set up SSH access for PyPI. The procedure works like this:
>>>>
>>>> Sorry if this should be obvious, but what does ssh access to PyPI give
>>>> me, as a developer?  Can I manage uploads that way or something?
>>>
>>> In principle, you would be able to use all PyPI-related distutils
>>> commands (register, upload) over SSH, which then would free you from
>>> the need to provide a password (interactively, or on disk).
>>
>> Ah, nice!
>>
>>> In practice, this would require distutils to be modified to actually
>>> use that protocol, instead of using http over plain TCP.
>>
>> Are those changes planned?
> 
> I can add this to Distutils' upload command (and Distribute until
> Distutils is released standalone)
> as soon as people have tried it manually successfully.

I'm afraid that's hardly possible without more background information.

The server doesn't return anything for an HTTP request and Martin
didn't provide an example of what to enter...

$ ssh -T submit at pypi.python.org
The authenticity of host 'pypi.python.org (82.94.164.163)' can't be established.
RSA key fingerprint is 13:6a:a5:0c:e0:3e:56:81:2f:13:d9:9f:86:5d:ab:6f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'pypi.python.org,82.94.164.163' (RSA) to the list of known hosts.
GET / HTTP/1.1

$

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 12 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From martin at v.loewis.de  Tue Jan 12 22:27:51 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 22:27:51 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <F08ABF9E-FCD4-4D1B-BAAC-4CF70787D471@gmail.com>
References: <4B4BA0DC.5060209@v.loewis.de>
	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>
	<4B4C128C.6070004@v.loewis.de>
	<F08ABF9E-FCD4-4D1B-BAAC-4CF70787D471@gmail.com>
Message-ID: <4B4CE957.4060606@v.loewis.de>

>> In practice, this would require distutils to be modified to actually
>> use that protocol, instead of using http over plain TCP.
> 
> Are those changes planned?

I personally don't have any plans, but I think patches are welcome
(atleast to distutils and distribute).

Regards,
Martin

From martin at v.loewis.de  Tue Jan 12 22:28:25 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 22:28:25 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <4B4C784F.8000705@egenix.com>
References: <4B4BA0DC.5060209@v.loewis.de> <4B4C784F.8000705@egenix.com>
Message-ID: <4B4CE979.9010300@v.loewis.de>

> Could you please summarize why SSH access to PyPI may be a
> useful feature to have and what the intended use is ?

See my message to Doug.

Regards,
Martin

From martin at v.loewis.de  Tue Jan 12 22:30:37 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 22:30:37 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <AC944FA0-9EC3-4E56-B4C9-44FE68FCAE69@gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>
	<4B4C13F8.5090707@v.loewis.de>
	<AC944FA0-9EC3-4E56-B4C9-44FE68FCAE69@gmail.com>
Message-ID: <4B4CE9FD.5070706@v.loewis.de>

> Is it being considered as the starting point, or not?

Not by me, no. I think meta-data mirroring is completely pointless
in order to achieve PEP 381's objective. Read the PEP, it specifies
fairly clearly what exactly needs to be mirrored.

Regards,
Martin

From martin at v.loewis.de  Tue Jan 12 22:31:24 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 22:31:24 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <94bdd2611001120539gbcb4965q9701c5ec906191de@mail.gmail.com>
References: <4B4BA0DC.5060209@v.loewis.de>	
	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>	
	<4B4C128C.6070004@v.loewis.de>	
	<F08ABF9E-FCD4-4D1B-BAAC-4CF70787D471@gmail.com>
	<94bdd2611001120539gbcb4965q9701c5ec906191de@mail.gmail.com>
Message-ID: <4B4CEA2C.10401@v.loewis.de>

> 
> I can add this to Distutils' upload command (and Distribute until
> Distutils is released standalone)
> as soon as people have tried it manually successfully.

This should probably affect both register and upload.

Regards,
Martin


From martin at v.loewis.de  Tue Jan 12 22:33:56 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 22:33:56 +0100
Subject: [Catalog-sig] PyPI ssh access
In-Reply-To: <4B4C7F9D.5070703@egenix.com>
References: <4B4BA0DC.5060209@v.loewis.de>	<9151B54C-61C2-41E6-BC61-4D0FA03E736B@gmail.com>	<4B4C128C.6070004@v.loewis.de>	<F08ABF9E-FCD4-4D1B-BAAC-4CF70787D471@gmail.com>
	<94bdd2611001120539gbcb4965q9701c5ec906191de@mail.gmail.com>
	<4B4C7F9D.5070703@egenix.com>
Message-ID: <4B4CEAC4.1000600@v.loewis.de>

> $ ssh -T submit at pypi.python.org
> The authenticity of host 'pypi.python.org (82.94.164.163)' can't be established.
> RSA key fingerprint is 13:6a:a5:0c:e0:3e:56:81:2f:13:d9:9f:86:5d:ab:6f.
> Are you sure you want to continue connecting (yes/no)? yes
> Warning: Permanently added 'pypi.python.org,82.94.164.163' (RSA) to the list of known hosts.
> GET / HTTP/1.1

Try the same URLs as through http, i.e. starting with /pypi.

For some reason, it doesn't issue a redirect on /, probably
because that is implemented in Apache, not in PyPI (I didn't
check).

Regards,
Martin

From ssteinerx at gmail.com  Tue Jan 12 23:01:46 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 12 Jan 2010 17:01:46 -0500
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4CE9FD.5070706@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>
	<4B4C13F8.5090707@v.loewis.de>
	<AC944FA0-9EC3-4E56-B4C9-44FE68FCAE69@gmail.com>
	<4B4CE9FD.5070706@v.loewis.de>
Message-ID: <47C4DB3B-8F81-4138-961F-50F9D76AB413@gmail.com>


On Jan 12, 2010, at 4:30 PM, Martin v. L?wis wrote:

>> Is it being considered as the starting point, or not?
> 
> Not by me, no. I think meta-data mirroring is completely pointless in order to achieve PEP 381's objective.

I don't quite understand that statement; the meta-data is part of the mirror.  z3c.pypimirror doesn't just mirror meta-data anyway, it does a full mirror (sort of).

If z3c.pypimirror is not the starting point then is there a different implementation that will be used as a 'jumpstart'?  The mirroring protocol is "XXX Need to describe the protocol here." and z3c.pypimirror is mentioned in the only other sentence within that section.

> Read the PEP, it specifies fairly clearly what exactly needs to be mirrored.

I have read it.  I'm mirroring the meta-data in order to make an easy way for tools that want to pull the meta-data to do it without beating the crap out of the xmlrpc interface on pypi.  This little project came out of a discussion on distutils around December 27th-30th.  (http://mail.python.org/pipermail/distutils-sig/2009-December/015178.html, http://mail.python.org/pipermail/distutils-sig/2009-December/015177.html).

I was asking because I thought it would be smart to just use and possibly improve the "sample code" for PEP 381 rather than start from scratch.

S


From martin at v.loewis.de  Tue Jan 12 23:20:45 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 12 Jan 2010 23:20:45 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <47C4DB3B-8F81-4138-961F-50F9D76AB413@gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>
	<4B4C13F8.5090707@v.loewis.de>
	<AC944FA0-9EC3-4E56-B4C9-44FE68FCAE69@gmail.com>
	<4B4CE9FD.5070706@v.loewis.de>
	<47C4DB3B-8F81-4138-961F-50F9D76AB413@gmail.com>
Message-ID: <4B4CF5BD.3060501@v.loewis.de>

>>> Is it being considered as the starting point, or not?
>> Not by me, no. I think meta-data mirroring is completely pointless
>> in order to achieve PEP 381's objective.
> 
> I don't quite understand that statement; the meta-data is part of the
> mirror.

Only if you apply a textbook definition of the word "mirror". For what
PEP 381 calls mirroring, you don't need the meta-data at all.

> If z3c.pypimirror is not the starting point then is there a different
> implementation that will be used as a 'jumpstart'?

I'll write it from scratch, using urllib (or perhaps Twisted). We
evalutated z3c.pypimirror at the last PyCon, and I came to the
conclusion that it is not appropriate as a starting point: it does
too little (not mirroring everything that PEP 381 tells it to), and
it does too much (also grabbing packages/distributions/parcels
from other places - i.e. not from PyPI), to support offline operation
(which is not the objective of PEP 381).

> I have read it.  I'm mirroring the meta-data in order to make an easy
> way for tools that want to pull the meta-data to do it without
> beating the crap out of the xmlrpc interface on pypi.

But you don't need that. All you need is the list of project names
to start with, and then the only XML-RPC operation will be to look
at the changelog. You don't need any of the other package metadata
(I don't quite recall why "pypi" is listed as a URL to mirror; IMO,
it shouldn't).

Regards,
Martin

From ziade.tarek at gmail.com  Tue Jan 12 23:22:53 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 12 Jan 2010 23:22:53 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <47C4DB3B-8F81-4138-961F-50F9D76AB413@gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>
	<4B4C13F8.5090707@v.loewis.de>
	<AC944FA0-9EC3-4E56-B4C9-44FE68FCAE69@gmail.com>
	<4B4CE9FD.5070706@v.loewis.de>
	<47C4DB3B-8F81-4138-961F-50F9D76AB413@gmail.com>
Message-ID: <94bdd2611001121422i5883f416kc7739460072aaed5@mail.gmail.com>

On Tue, Jan 12, 2010 at 11:01 PM, ssteinerX at gmail.com
<ssteinerx at gmail.com> wrote:
[..]
> If z3c.pypimirror is not the starting point then is there a different implementation that will be used as a 'jumpstart'? ?The mirroring protocol is "XXX Need to describe the protocol here." and z3c.pypimirror is mentioned in the only other sentence within that section.

The "XXX Need to describe the protocol here." I've added is just
because I didn't take the time yet to write down the implicit protocol
that was defined some years ago and that z3c.pypimirror uses. The
objective here
is to describe the most efficient way to scan and look for things to
mirror (via the changelog) etc.


Tarek

From ziade.tarek at gmail.com  Tue Jan 12 23:24:54 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 12 Jan 2010 23:24:54 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4CF5BD.3060501@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<FA7C534C-012D-405E-9C29-6495518BF189@gmail.com>
	<4B4C13F8.5090707@v.loewis.de>
	<AC944FA0-9EC3-4E56-B4C9-44FE68FCAE69@gmail.com>
	<4B4CE9FD.5070706@v.loewis.de>
	<47C4DB3B-8F81-4138-961F-50F9D76AB413@gmail.com>
	<4B4CF5BD.3060501@v.loewis.de>
Message-ID: <94bdd2611001121424h7e61e0dbxbe1a7f377cf3f2a2@mail.gmail.com>

On Tue, Jan 12, 2010 at 11:20 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
[..]
>
> But you don't need that. All you need is the list of project names
> to start with, and then the only XML-RPC operation will be to look
> at the changelog. You don't need any of the other package metadata
> (I don't quite recall why "pypi" is listed as a URL to mirror; IMO,
> it shouldn't).

That's not to be mirrored indeed, I'll change this.

From ssteinerx at gmail.com  Wed Jan 13 00:04:55 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 12 Jan 2010 18:04:55 -0500
Subject: [Catalog-sig] PEP 381 Sample/Reference Implementation,
	Mirroring PyPI Meta Data
Message-ID: <7F780315-2C82-41EE-91DF-7AB4A70AB78D@gmail.com>

I've taken this off of the "PyPI down" thread (sorry, Chris).

I was poking at a little project that came out of this discussion on distutils around December 27th-30th.  http://mail.python.org/pipermail/distutils-sig/2009-December/015178.html
http://mail.python.org/pipermail/distutils-sig/2009-December/015177.html

Essentially, the goal is/was to put a pkl of all of the meta data up so that someone wanting to mirror just the metadata (for whatever reason) could start with an up-to-date set, then just pull deltas from pypi.

I thought it'd be smart to just use and possibly improve the "sample code" for PEP 381 rather than start from scratch.

Now I have:

On Jan 12, 2010, at 5:20 PM, Martin v. L?wis wrote:

>> If z3c.pypimirror is not the starting point then is there a different
>> implementation that will be used as a 'jumpstart'?
> 
> I'll write it from scratch, using urllib (or perhaps Twisted). We
> evalutated z3c.pypimirror at the last PyCon, and I came to the
> conclusion that it is not appropriate as a starting point: it does
> too little (not mirroring everything that PEP 381 tells it to), and
> it does too much (also grabbing packages/distributions/parcels
> from other places - i.e. not from PyPI), to support offline operation
> (which is not the objective of PEP 381).

And:

On Jan 11, 2010, at 4:22 PM, Tarek Ziad? wrote:

> If we have time at Pycon, and if some folks are interested, I would
> like to finish the implementation of PEP 381,

On Jan 12, 2010, at 5:22 PM, Tarek Ziad? wrote:

>> If z3c.pypimirror is not the starting point then is there a different implementation that will be used as a 'jumpstart'?  The mirroring protocol is "XXX Need to describe the protocol here." and z3c.pypimirror is mentioned in the only other sentence within that section.
> 
> The "XXX Need to describe the protocol here." I've added is just because I didn't take the time yet to write down the implicit protocol that was defined some years ago and that z3c.pypimirror uses. The objective here is to describe the most efficient way to scan and look for things tomirror (via the changelog) etc.

So...it seems that Tarek thinks the z3c.pypimirror implements the eventual protocol and Martin is just planning on writing "it" from scratch himself.

If this is to be a group effort, perhaps this would be a good time to whack up a repository with whatever specification (beyond what's in the PEP) or starting-point code is available and I'll do my little project within that framework.  Or not.

Tarek?  Martin?

S

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100112/7905385e/attachment.htm>

From martin at v.loewis.de  Wed Jan 13 00:35:01 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 13 Jan 2010 00:35:01 +0100
Subject: [Catalog-sig] PEP 381 Sample/Reference Implementation,
 Mirroring PyPI Meta Data
In-Reply-To: <7F780315-2C82-41EE-91DF-7AB4A70AB78D@gmail.com>
References: <7F780315-2C82-41EE-91DF-7AB4A70AB78D@gmail.com>
Message-ID: <4B4D0725.1090808@v.loewis.de>

> So...it seems that Tarek thinks the z3c.pypimirror implements the
> eventual protocol and Martin is just planning on writing "it" from
> scratch himself.

Yes, Tarek is probably not convinced that z3c.pypimirror is not a
good starting point. I may be actually wrong in assuming that it is
better to start from scratch.

> If this is to be a group effort, perhaps this would be a good time to
> whack up a repository with whatever specification (beyond what's in the
> PEP) or starting-point code is available and I'll do my little project
> within that framework.  Or not.
> 
> Tarek?  Martin?

I don't have any code to share, beyond what is already implemented in
PyPI to support PEP 381 (and which is Tarek's work largely).

As for the protocol, I think the PEP says it all.

Regards,
Martin

From ssteinerx at gmail.com  Wed Jan 13 01:18:06 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 12 Jan 2010 19:18:06 -0500
Subject: [Catalog-sig] PEP 381 Sample/Reference Implementation,
	Mirroring PyPI Meta Data
In-Reply-To: <4B4D0725.1090808@v.loewis.de>
References: <7F780315-2C82-41EE-91DF-7AB4A70AB78D@gmail.com>
	<4B4D0725.1090808@v.loewis.de>
Message-ID: <A17A1383-FD6B-4D6A-A3B8-92DEC640D697@gmail.com>


On Jan 12, 2010, at 6:35 PM, Martin v. L?wis wrote:

>> So...it seems that Tarek thinks the z3c.pypimirror implements the
>> eventual protocol and Martin is just planning on writing "it" from
>> scratch himself.
> 
> Yes, Tarek is probably not convinced that z3c.pypimirror is not a
> good starting point. I may be actually wrong in assuming that it is
> better to start from scratch.
> 
>> If this is to be a group effort, perhaps this would be a good time to
>> whack up a repository with whatever specification (beyond what's in the
>> PEP) or starting-point code is available and I'll do my little project
>> within that framework.  Or not.
>> 
>> Tarek?  Martin?
> 
> I don't have any code to share, beyond what is already implemented in
> PyPI to support PEP 381 (and which is Tarek's work largely).
> 
> As for the protocol, I think the PEP says it all.

I'm sure this is clever but it must just be too subtle for me.

Uh...Tarek?

S


From ben+python at benfinney.id.au  Wed Jan 13 01:29:19 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 13 Jan 2010 11:29:19 +1100
Subject: [Catalog-sig] [Distutils]  packaging terminology confusion
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
	<3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
	<C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>
	<65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>
	<1289.218.214.45.58.1263254993.squirrel@syd-srv02.ezyreg.com>
	<87k4voqf00.fsf@benfinney.id.au>
	<1640.218.214.45.58.1263260689.squirrel@syd-srv02.ezyreg.com>
Message-ID: <87my0iq23k.fsf@benfinney.id.au>

"David Lyon" <david.lyon at preisshare.net> writes:

> If you have a zip/archive file, you can put anything in it. No reason
> why 'everything' can't go in it.
>
> A "L'Oeuf incredible" might include a Python 2.x and Python 3.x code
> set, make code for linux, .pyd for windows.
>
> It would be so un-confusing to have an egg like that.

I disagree entirely with that last statement.

However, you're now talking about changing the package format, and not
the terminology of what we already have. So I'm dropping this
sub-thread.

-- 
 \            ?Simplicity is prerequisite for reliability.? ?Edsger W. |
  `\                                                          Dijkstra |
_o__)                                                                  |
Ben Finney


From ziade.tarek at gmail.com  Wed Jan 13 01:31:47 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 13 Jan 2010 01:31:47 +0100
Subject: [Catalog-sig] PEP 381 Sample/Reference Implementation,
	Mirroring 	PyPI Meta Data
In-Reply-To: <A17A1383-FD6B-4D6A-A3B8-92DEC640D697@gmail.com>
References: <7F780315-2C82-41EE-91DF-7AB4A70AB78D@gmail.com>
	<4B4D0725.1090808@v.loewis.de>
	<A17A1383-FD6B-4D6A-A3B8-92DEC640D697@gmail.com>
Message-ID: <94bdd2611001121631n306562bfo5207a1d1d59793b2@mail.gmail.com>

2010/1/13 ssteinerX at gmail.com <ssteinerx at gmail.com>:
>
> On Jan 12, 2010, at 6:35 PM, Martin v. L?wis wrote:
>
>>> So...it seems that Tarek thinks the z3c.pypimirror implements the
>>> eventual protocol and Martin is just planning on writing "it" from
>>> scratch himself.
>>
>> Yes, Tarek is probably not convinced that z3c.pypimirror is not a
>> good starting point. I may be actually wrong in assuming that it is
>> better to start from scratch.

Frankly I don't know.  z3c.pypimirror provides the piece of code that
browses the changelog, and that
can be reused. But there's more stuff to do to implement 381, and it
depends a lot on the
implementation strategy : a mirror is a web application so they are a
thousand ways to implement it. In particular in the way to publish
download hits. PyPI scans Apache logs to sums them.

[..]
>> As for the protocol, I think the PEP says it all.
>
> I'm sure this is clever but it must just be too subtle for me.
>
> Uh...Tarek?

Are you coming at Pycon ? I can show you what we've done and the
principles. Or drop by irc

Tarek

-- 
Tarek Ziad? | http://ziade.org

From david.lyon at preisshare.net  Wed Jan 13 01:51:56 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 13 Jan 2010 11:51:56 +1100 (EST)
Subject: [Catalog-sig] [Distutils] packaging terminology confusion
In-Reply-To: <87my0iq23k.fsf@benfinney.id.au>
References: <4957f1ef1001070340y94da3cbgbc6dfe6c05028f3f@mail.gmail.com>
	<94bdd2611001071339g15942eb3rcf00dcb8ddd4ee5e@mail.gmail.com>
	<4B46576F.7070309@egenix.com>
	<4957f1ef1001071357y382db993qffaee3b5ba2277e2@mail.gmail.com>
	<4B465C5F.7030609@egenix.com>
	<94bdd2611001071422p56182196mb974e747a0adecd2@mail.gmail.com>
	<65e0bb521001071943n7bbf739au3f9c7fc9175c6d01@mail.gmail.com>
	<723929E9-9814-4005-9025-4F38355DB5E7@twistedmatrix.com>
	<3282.218.214.45.58.1263171427.squirrel@syd-srv02.ezyreg.com>
	<C0488BE7-A339-4D2B-B240-0BD1D71ECE79@python.org>
	<65e0bb521001110717i3f3b8941t1048eb71b2789686@mail.gmail.com>
	<1289.218.214.45.58.1263254993.squirrel@syd-srv02.ezyreg.com>
	<87k4voqf00.fsf@benfinney.id.au>
	<1640.218.214.45.58.1263260689.squirrel@syd-srv02.ezyreg.com>
	<87my0iq23k.fsf@benfinney.id.au>
Message-ID: <1479.218.214.45.58.1263343916.squirrel@syd-srv02.ezyreg.com>

> Ben  writes:

> However, you're now talking about changing the package format, and not
> the terminology of what we already have. So I'm dropping this
> sub-thread.

ok - up to you. Don't talk about:

  http://www.python.org/dev/peps/pep-0376/

David



From ssteinerx at gmail.com  Wed Jan 13 02:04:43 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 12 Jan 2010 20:04:43 -0500
Subject: [Catalog-sig] PEP 381 Sample/Reference Implementation,
	Mirroring  PyPI Meta Data
In-Reply-To: <94bdd2611001121631n306562bfo5207a1d1d59793b2@mail.gmail.com>
References: <7F780315-2C82-41EE-91DF-7AB4A70AB78D@gmail.com>
	<4B4D0725.1090808@v.loewis.de>
	<A17A1383-FD6B-4D6A-A3B8-92DEC640D697@gmail.com>
	<94bdd2611001121631n306562bfo5207a1d1d59793b2@mail.gmail.com>
Message-ID: <B90E4D76-6C08-4463-8A22-68DC68B85E67@gmail.com>


On Jan 12, 2010, at 7:31 PM, Tarek Ziad? wrote:

> 2010/1/13 ssteinerX at gmail.com <ssteinerx at gmail.com>:
>> 
>> On Jan 12, 2010, at 6:35 PM, Martin v. L?wis wrote:
>> 
>>>> So...it seems that Tarek thinks the z3c.pypimirror implements the
>>>> eventual protocol and Martin is just planning on writing "it" from
>>>> scratch himself.
>>> 
>>> Yes, Tarek is probably not convinced that z3c.pypimirror is not a
>>> good starting point. I may be actually wrong in assuming that it is
>>> better to start from scratch.
> 
> Frankly I don't know.  z3c.pypimirror provides the piece of code that
> browses the changelog, and that can be reused. But there's more stuff to do to implement 381, and it
> depends a lot on the implementation strategy : a mirror is a web application so they are a
> thousand ways to implement it. In particular in the way to publish
> download hits. PyPI scans Apache logs to sums them.

Yes, there are a million details and what z3c.pypimirror does is, basically, use the xmlrpc interface to get a list of what to process, then slogs its way through that way to figure out what to "mirror."  Not very pretty (or efficient).

> [..]
>>> As for the protocol, I think the PEP says it all.
>> 
>> I'm sure this is clever but it must just be too subtle for me.
>> 
>> Uh...Tarek?
> 
> Are you coming at Pycon ? I can show you what we've done and the
> principles. Or drop by irc

I'm working on making it to Pycon, we can chat on irc tomorrow.

S


From sridharr at activestate.com  Wed Jan 13 07:27:23 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Tue, 12 Jan 2010 22:27:23 -0800
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <4B4BFC89.6060701@activestate.com>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B8C5D.3040409@v.loewis.de>	<4B4B8CFF.9050505@simplistix.co.uk>	<hig65h$d58$1@ger.gmane.org>
	<4B4BFC89.6060701@activestate.com>
Message-ID: <4B4D67CB.6030104@activestate.com>

On 1/11/2010 8:37 PM, Sridhar Ratnakumar wrote:
> [1] apologies for not setting the user-agent string properly; I will get
> around to doing it tomorrow

Done. The User-Agent string (setuptools.package_index.user_agent) will 
have "PyPM" appended to it.

-srid

From jmg3000 at gmail.com  Thu Jan 14 16:27:14 2010
From: jmg3000 at gmail.com (John Gabriele)
Date: Thu, 14 Jan 2010 10:27:14 -0500
Subject: [Catalog-sig] policies for not stepping on other package's toes?
Message-ID: <65e0bb521001140727i95e554dicb985cf4ffb876b3@mail.gmail.com>

Hi,

Does the PyPI allow two differently-named distributions to contain
identically-named packages? For example, if MyStuff-1.0.0.tgz contains
a `mypackage/foo.py` module, does PyPI mind if CoolStuff-2.4.1.tgz
contains a `mypackage/bar.py` module?

How about if CoolStuff-2.4.1.tgz contains a `mypackage/foo.py` module
(which would conflict with the one packaged by MyStuff)?

Does the PyPI allow two differently-named distributions to contain
to-be-installed scripts (as specified by the `scripts=[...]` line in
the setup() call) with the same name? That is, If MyStuff-1.0.0.tgz
contains and will install a `do-stuff.py` script into the user's bin
dir, and someone else's CoolStuff-2.4.1.tgz *also* contains and will
install a `do-stuff.py` script, will the PyPI catch this and disallow
it?

Thanks,
---John

From hanno at hannosch.eu  Thu Jan 14 16:56:39 2010
From: hanno at hannosch.eu (Hanno Schlichting)
Date: Thu, 14 Jan 2010 16:56:39 +0100
Subject: [Catalog-sig] policies for not stepping on other package's toes?
In-Reply-To: <65e0bb521001140727i95e554dicb985cf4ffb876b3@mail.gmail.com>
References: <65e0bb521001140727i95e554dicb985cf4ffb876b3@mail.gmail.com>
Message-ID: <5cae42b21001140756r53effe58t44d37aa26e38cb94@mail.gmail.com>

On Thu, Jan 14, 2010 at 4:27 PM, John Gabriele <jmg3000 at gmail.com> wrote:
> Does the PyPI allow two differently-named distributions to contain
> identically-named packages?

Sure, it does. PyPi only guarantees the uniqueness of the distribution
name. And it can only do that for projects hosted on PyPi, there's
nothing preventing anyone to host a distribution with the same name on
a different service.

As an obvious example take distribute / setuptools. One being a fork
of the other, means they currently contain packages and scripts with
the exact same names.

There's also cases where projects change their distribution formats
after a while. Like the Chameleon distribution, which now contains the
code formerly found in the separate chameleon.core and chameleon.zpt
distributions but the package structure stays the same.

Hanno

From ssteinerx at gmail.com  Sat Jan 16 03:05:47 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Fri, 15 Jan 2010 21:05:47 -0500
Subject: [Catalog-sig] PyPI XML-RPC multicall support
In-Reply-To: <4B390700.9030100@v.loewis.de>
References: <4B390700.9030100@v.loewis.de>
Message-ID: <66110C7D-6A21-4656-80B5-CD4DA6D4B4CC@gmail.com>


On Dec 28, 2009, at 2:29 PM, Martin v. L?wis wrote:

> The PyPI XML-RPC server now supports multicalls, with a limit
> of 100 subcalls per multicall. So if you have an XML-RPC
> application that makes thousands of calls to PyPI in a short
> time, consider batching them. I was able to obtain a speedup
> of four in an application that tries to get all release data
> and urls.

Hey!  Don't know if anyone else cares, but I just added this to my little metadata mirror program to batch up all the releases for each package into one call and it made a huge difference.  I didn't bother to measure/time anything (it's already taking long enough!), but the improvement is noticeable as I watch the packages fly by.

Still trying to get one full set of data as my early attempts were, shall we say, not handling interruptions well...

BTW, could you look in the logs real quick and see if my program's User-Agent "pypimeta-mirror" is coming through properly on your end?

Thanks,

Steve

aka/ssteinerX
aka/Steve Steiner


From martin at v.loewis.de  Sun Jan 17 10:43:19 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 17 Jan 2010 10:43:19 +0100
Subject: [Catalog-sig] PyPI XML-RPC multicall support
In-Reply-To: <66110C7D-6A21-4656-80B5-CD4DA6D4B4CC@gmail.com>
References: <4B390700.9030100@v.loewis.de>
	<66110C7D-6A21-4656-80B5-CD4DA6D4B4CC@gmail.com>
Message-ID: <4B52DBB7.8060605@v.loewis.de>

> BTW, could you look in the logs real quick and see if my program's
> User-Agent "pypimeta-mirror" is coming through properly on your end?

Take a look at

http://pypi.python.org/webstats/usage_201001.html

Regards,
Martin


From ssteinerx at gmail.com  Sun Jan 17 13:38:53 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Sun, 17 Jan 2010 07:38:53 -0500
Subject: [Catalog-sig] PyPI XML-RPC multicall support
In-Reply-To: <4B52DBB7.8060605@v.loewis.de>
References: <4B390700.9030100@v.loewis.de>
	<66110C7D-6A21-4656-80B5-CD4DA6D4B4CC@gmail.com>
	<4B52DBB7.8060605@v.loewis.de>
Message-ID: <DEB0DC89-A986-410F-A251-0D6AEAC0AFE8@gmail.com>

 
On Jan 17, 2010, at 4:43 AM, Martin v. L?wis wrote:

>> BTW, could you look in the logs real quick and see if my program's
>> User-Agent "pypimeta-mirror" is coming through properly on your end?
> 
> Take a look at
> 
> http://pypi.python.org/webstats/usage_201001.html

Awesome, thank you!  I'll add an appropriate version number when I get something releasable.

Right now, I've got a full set of data, in an SQLITE table (using xmlrpc MultiCall; boy did that speed things up!), and am working on writing that out to a pkl, and doing a quick API class for anyone to play around with the data.  

There sure is a lot of data when you get all releases.  Some have more than 50 releases (I know 'cause it blew up the MultiCall interface)

I won't be doing any more full pulls (unless something is horribly wrong with my stored data).  Hopefully, now that I'm a little more up to speed, I'll be able to contribute a little more intelligently to the PEP 381 discussions.

S




From jannis at leidel.info  Mon Jan 18 17:27:29 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 17:27:29 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
Message-ID: <420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>

Am 11.01.2010 um 22:22 schrieb Tarek Ziad?:
>> Yup, nothing out of the ordinary... oh well, I guess it is just me and I'll
>> wait and hope things get better ;-)
> 
> If we have time at Pycon, and if some folks are interested, I would
> like to finish the implementation of PEP 381,

FWIW, I don't see the PEP to be completed. The actual mirror protocol, how to handle multiple (unsynchronized) indexes and the API design are clearly undecided -- and not discussed.

> and if some folks in the Pip team are interested, see if we can
> develop a clisent-side geolocalisation
> thingie to pick the closest PyPI mirror.

Sadly I won't make it to this year's Pycon, even though I would be very interested in adding mirror support to pip (beyond --find-links and --index).

Jannis


From ziade.tarek at gmail.com  Mon Jan 18 18:25:47 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 18 Jan 2010 18:25:47 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
Message-ID: <94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>

On Mon, Jan 18, 2010 at 5:27 PM, Jannis Leidel <jannis at leidel.info> wrote:
> Am 11.01.2010 um 22:22 schrieb Tarek Ziad?:
>>> Yup, nothing out of the ordinary... oh well, I guess it is just me and I'll
>>> wait and hope things get better ;-)
>>
>> If we have time at Pycon, and if some folks are interested, I would
>> like to finish the implementation of PEP 381,
>
> FWIW, I don't see the PEP to be completed. The actual mirror protocol, how to handle multiple (unsynchronized) indexes and the API design are clearly undecided -- and not discussed.

If by "unsynchronized" you mean handling mutiple indexes one client
side that are not mirrors, that's not a protocol the PEP defines.
(even though it gives some info on it)
This is the business of the client software, and there's no API design
required for that in this PEP.

The PEP defines the mirroring protocol and the pages to be maintained
in a PyPI like server.


>
>> and if some folks in the Pip team are interested, see if we can
>> develop a clisent-side geolocalisation
>> thingie to pick the closest PyPI mirror.
>
> Sadly I won't make it to this year's Pycon,
>even though I would be very interested in adding mirror support to pip (beyond --find-links and --index).

Too bad :/

From jannis at leidel.info  Mon Jan 18 18:47:45 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 18:47:45 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
Message-ID: <0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>


Am 18.01.2010 um 18:25 schrieb Tarek Ziad?:

> On Mon, Jan 18, 2010 at 5:27 PM, Jannis Leidel <jannis at leidel.info> wrote:
>> Am 11.01.2010 um 22:22 schrieb Tarek Ziad?:
>>>> Yup, nothing out of the ordinary... oh well, I guess it is just me and I'll
>>>> wait and hope things get better ;-)
>>> 
>>> If we have time at Pycon, and if some folks are interested, I would
>>> like to finish the implementation of PEP 381,
>> 
>> FWIW, I don't see the PEP to be completed. The actual mirror protocol, how to handle multiple (unsynchronized) indexes and the API design are clearly undecided -- and not discussed.
> 
> If by "unsynchronized" you mean handling mutiple indexes one client
> side that are not mirrors, that's not a protocol the PEP defines.
> (even though it gives some info on it)
> This is the business of the client software, and there's no API design
> required for that in this PEP.

With "unsychronized" I mean indexes that are registered with PyPI to mirror it, but aren't up-to-date because of loss of connectivity/failure/interest.

Will the mirrors check periodically the main index for updates?
Or will PyPI send out a message to the slaves, requesting an update, e.g. with WebHooks [1] or PubSubHubbub [2]?
Is this API going to be open for other non-official/non-open mirrors?

> The PEP defines the mirroring protocol and the pages to be maintained
> in a PyPI like server.

Oh, you mean it will define it? Since there is just a placeholder on [3] :)

Jannis


1: http://wiki.webhooks.org/
2: http://en.wikipedia.org/wiki/PubSubHubbub
3: http://www.python.org/dev/peps/pep-0381/#the-mirroring-protocol


From ziade.tarek at gmail.com  Mon Jan 18 19:03:15 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 18 Jan 2010 19:03:15 +0100
Subject: [Catalog-sig] PyPI down?
In-Reply-To: <0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>
Message-ID: <94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>

On Mon, Jan 18, 2010 at 6:47 PM, Jannis Leidel <jannis at leidel.info> wrote:
[..]
>
> With "unsychronized" I mean indexes that are registered with PyPI to mirror it, but aren't up-to-date because of loss of connectivity/failure/interest.

So in  that case pip will just have to check for the last modified
date, as explained in the PEP, to know how "fresh" a mirror is. The
strategy to take depending on this freshness it's up to the client
program.

> Will the mirrors check periodically the main index for updates?

Mirrors are dealing on their own to be up-to-date, by checking PyPI
periodically.

> Or will PyPI send out a message to the slaves, requesting an update, e.g. with WebHooks [1] or PubSubHubbub [2]?

No, the only event that will happen is the daily call the PyPI server
will do on each mirror to get the stats. But that's unrelated to the
mirror being updated.

> Is this API going to be open for other non-official/non-open mirrors?

Which API ?

>> The PEP defines the mirroring protocol and the pages to be maintained
>> in a PyPI like server.
>
> Oh, you mean it will define it? Since there is just a placeholder on [3] :)

As I said earlier, the mirroring (or should I say the 'fetch')
protocol is already defined and in usage for quite a time
(easy_install, z3c.pypimirror). Here it's just a matter of writing it
down here.

The rest of the PEP is just defining standard places/formats for stats
files and last modified dates.

Tarek

From ziade.tarek at gmail.com  Mon Jan 18 19:11:47 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 18 Jan 2010 19:11:47 +0100
Subject: [Catalog-sig] Setting up the Reply-To field in mailman
Message-ID: <94bdd2611001181011w4bb3a58dt7b47eafaf197da9a@mail.gmail.com>

Hi,

Could we set the "Reply-To" field to the email of the mailing-list ?

This is quite useful to avoid long CC lists and to avoid forgetting to
CC the mailing-list

(because we all hit "reply-all" to make sure the Mailing-list receives
the answer, and
not only the person who sent the initial mail)

Thanks
Tarek
-- 
Tarek Ziad? | http://ziade.org

From jannis at leidel.info  Mon Jan 18 19:29:59 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 19:29:59 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>
	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>
Message-ID: <BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>

Changing this subject..

Am 18.01.2010 um 19:03 schrieb Tarek Ziad?:

> On Mon, Jan 18, 2010 at 6:47 PM, Jannis Leidel <jannis at leidel.info> wrote:
> [..]
>> With "unsychronized" I mean indexes that are registered with PyPI to mirror it, but aren't up-to-date because of loss of connectivity/failure/interest.
> 
> So in  that case pip will just have to check for the last modified
> date, as explained in the PEP, to know how "fresh" a mirror is. The
> strategy to take depending on this freshness it's up to the client
> program.

Really, pip gets to decide which package is fresh enough? Like a PIP_BEST_BEFORE setting var?

>> Will the mirrors check periodically the main index for updates?
> Mirrors are dealing on their own to be up-to-date, by checking PyPI
> periodically.
>> Or will PyPI send out a message to the slaves, requesting an update, e.g. with WebHooks [1] or PubSubHubbub [2]?
> 
> No, the only event that will happen is the daily call the PyPI server
> will do on each mirror to get the stats. But that's unrelated to the
> mirror being updated.
>> Is this API going to be open for other non-official/non-open mirrors?
> Which API ?

The API of PyPI which would actively ping mirrors to update their package data. Those "webhooks" are already provided by project hosting sites like Google Code, Github and Bitbucket for post-commit/post-push situations. 

I don't see a reason why PyPI shouldn't use a similar technique to let mirrors know when an update happened. It lowers the number of periodic requests to PyPI from the mirrors, simplifies the setup of slave mirrors and -- as a bonus -- makes the life of continuous integration projects like pony-build easier.

Jannis


From ziade.tarek at gmail.com  Mon Jan 18 19:49:38 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 18 Jan 2010 19:49:38 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>
	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>
	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>
Message-ID: <94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>

On Mon, Jan 18, 2010 at 7:29 PM, Jannis Leidel <jannis at leidel.info> wrote:
[..]
>> So in ?that case pip will just have to check for the last modified
>> date, as explained in the PEP, to know how "fresh" a mirror is. The
>> strategy to take depending on this freshness it's up to the client
>> program.
>
> Really, pip gets to decide which package is fresh enough? Like a PIP_BEST_BEFORE setting var?

Pip gets to decide which *mirror* to use, given their last modified
dates. I am not sure what would be the best strategy here.


>>> Is this API going to be open for other non-official/non-open mirrors?
>> Which API ?
>
> The API of PyPI which would actively ping mirrors to update their package data.

We v'e discussed this last year, and came up with the conclusion that
asking PyPI to actively call each mirror was quite an intensive work
because it means it has to call each mirror for each update (there are
many updates per hour), and deal with a timeout for each request, etc.
IOW, work *all day long* just for that.

The other problem is that when a mirror is down or unreachable for a
while, it can't get that ping. So what happens is that the mirror
still needs to update itself in these situations. (because PyPI will
certainly not implement a replay-system when some ping fails.)

So why bother setting up two different update systems ? each mirror
can look at the CHANGELOG every minute or so and get updated on their
side.

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From jannis at leidel.info  Mon Jan 18 20:19:52 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 20:19:52 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>
	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>
	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>
	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>
Message-ID: <79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>

Am 18.01.2010 um 19:49 schrieb Tarek Ziad?:

> On Mon, Jan 18, 2010 at 7:29 PM, Jannis Leidel <jannis at leidel.info> wrote:
> [..]
>>> So in  that case pip will just have to check for the last modified
>>> date, as explained in the PEP, to know how "fresh" a mirror is. The
>>> strategy to take depending on this freshness it's up to the client
>>> program.
>> 
>> Really, pip gets to decide which package is fresh enough? Like a PIP_BEST_BEFORE setting var?
> 
> Pip gets to decide which *mirror* to use, given their last modified
> dates. I am not sure what would be the best strategy here.
Interesting, that would require pip to fetch the last modified dates of all mirrors, I guess. Not sure if that'd be ideal.

>>>> Is this API going to be open for other non-official/non-open mirrors?
>>> Which API ?
>> 
>> The API of PyPI which would actively ping mirrors to update their package data.
> 
> We v'e discussed this last year, and came up with the conclusion that
> asking PyPI to actively call each mirror was quite an intensive work
> because it means it has to call each mirror for each update (there are
> many updates per hour), and deal with a timeout for each request, etc.
> IOW, work *all day long* just for that.

I'm not saying it's easier to implement (which I believe isn't the goal of a PEP anyway), only that it would give the "mirror" idea a little more meaning; more than to just spread the files across multiple servers.

> The other problem is that when a mirror is down or unreachable for a
> while, it can't get that ping. So what happens is that the mirror
> still needs to update itself in these situations. (because PyPI will
> certainly not implement a replay-system when some ping fails.)

Why not? The ping from PyPI to the mirrors would simply tell them to ask PyPI for updates since the last time they were updated. In case a ping doesn't reach a mirror it'll get updated next time it receives a ping.

> So why bother setting up two different update systems ? each mirror
> can look at the CHANGELOG every minute or so and get updated on their
> side.

I'm not proposing two update systems. IMO, there is a difference between the message "package was updated" and the actual mirroring of the package following that message. Each are most useful when combined of course, but the messaging shouldn't be limited to be used only by the mirroring.

Jannis


From ziade.tarek at gmail.com  Mon Jan 18 20:40:29 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 18 Jan 2010 20:40:29 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>
	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>
	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>
	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>
	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>
Message-ID: <94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>

On Mon, Jan 18, 2010 at 8:19 PM, Jannis Leidel <jannis at leidel.info> wrote:
[...]
>
> Why not? The ping from PyPI to the mirrors would simply tell them to ask PyPI for updates since the last time they were updated. In case a ping doesn't reach a mirror it'll get updated next time it receives a ping.

So in that case, a ping would not be specific to a project at PyPI
being updated, but just to notice that  CHANGELOG has changed ?

>
>> So why bother setting up two different update systems ? each mirror
>> can look at the CHANGELOG every minute or so and get updated on their
>> side.
>
> I'm not proposing two update systems. IMO, there is a difference between the message "package was updated" and the actual mirroring of the package following that message. Each are most useful when combined of course, but the messaging shouldn't be limited to be used only by the mirroring.

If PyPI calls other servers for something else than reading the stats,
it should be a call that returns instantly (with a very fast timeout
as well). In that case, I think it could be done technically.

But yet, I don't really see the use case: what is the big difference
of having PyPI ping you, let's say, ten times per hour, and you
looking if the changelog has changed once every ten minutes ?

What is the usage ? mirrors will always be a bit unsynchronized, since
a mirroring protocol is not a real-time synchronization system. A
one-hour lag is acceptable here.

Tarek

-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Mon Jan 18 21:47:04 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 18 Jan 2010 21:47:04 +0100
Subject: [Catalog-sig] PEP 381 (Was: PyPI down?)
In-Reply-To: <420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
Message-ID: <4B54C8C8.6050703@v.loewis.de>

> FWIW, I don't see the PEP to be completed. The actual mirror
> protocol, how to handle multiple (unsynchronized) indexes and the API
> design are clearly undecided -- and not discussed.

The actual mirror protocol *is* decided, even though it's not explicitly
spelled out in the PEP. It is based entirely on existing API; no new API
on the PyPI side is planned. Basically, you do a lot of HTTP GETs.

As Tarek explains, the handling of unsynchronized indices is also
discussed. Each mirror has a timestamp of last synchronization.

I expect that mirrors won't be behind more than two minutes. If you
have a strong requirement to provide a better quality, you'll need
to propose a PEP change. I would personally prefer to have such a
feature only in a future revision of the protocol, based on practical
experience.

If you would like to propose a push model, I'd be curious what the
advantage would be of one of the complicated-sounding APIs you mentioned
over a simple trigger URL that the mirror might provide.

Regards,
Martin

From martin at v.loewis.de  Mon Jan 18 21:57:56 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 18 Jan 2010 21:57:56 +0100
Subject: [Catalog-sig] Changes to the simple pages
Message-ID: <4B54CB54.9020001@v.loewis.de>

I would like to change the simple API in the following
ways, within a few days:

1. XML-escape href attributes to make the pages well-formed XML
   (possibly other changes required for that as well, but I only
    ran into lack of &amp;)
2. Make file paths relative (from http://pypi.python.org/packages
   to ../../packages)

Objections?

Regards,
Martin

From jannis at leidel.info  Mon Jan 18 22:29:38 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 22:29:38 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>
	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>
	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>
	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>
	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>
	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>
Message-ID: <C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>


Am 18.01.2010 um 20:40 schrieb Tarek Ziad?:

> On Mon, Jan 18, 2010 at 8:19 PM, Jannis Leidel <jannis at leidel.info> wrote:
> [...]
>> 
>> Why not? The ping from PyPI to the mirrors would simply tell them to ask PyPI for updates since the last time they were updated. In case a ping doesn't reach a mirror it'll get updated next time it receives a ping.
> 
> So in that case, a ping would not be specific to a project at PyPI
> being updated, but just to notice that  CHANGELOG has changed ?

Actually it doesn't matter if the ping says "project x updated" or "changelog of index updated". The only important information of that ping (or should I say HTTP POST request) is its timestamp. The mirror would be able to compare it to its own "last-updated" value to decide to mirror or what to do.

>>> So why bother setting up two different update systems ? each mirror
>>> can look at the CHANGELOG every minute or so and get updated on their
>>> side.
>> 
>> I'm not proposing two update systems. IMO, there is a difference between the message "package was updated" and the actual mirroring of the package following that message. Each are most useful when combined of course, but the messaging shouldn't be limited to be used only by the mirroring.
> 
> If PyPI calls other servers for something else than reading the stats,
> it should be a call that returns instantly (with a very fast timeout
> as well). In that case, I think it could be done technically.
> 
> But yet, I don't really see the use case: what is the big difference
> of having PyPI ping you, let's say, ten times per hour, and you
> looking if the changelog has changed once every ten minutes?

The big difference is the integration with other systems over an common technology (HTTP), mirroring being just one use case.

> What is the usage ? mirrors will always be a bit unsynchronized, since
> a mirroring protocol is not a real-time synchronization system. A
> one-hour lag is acceptable here.

Besides the reference use case "mirroring"?

I don't know. Let's think about potential web services that could use information about a new software release:

Continous integration, yay!
Social communities with voting and commenting, yay!
Version control, yay!
"Private" mirrors (as in "not advertised in the public")
http://pypants.org
http://heroku.com

and so on..

Jannis




From jannis at leidel.info  Mon Jan 18 22:29:45 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 22:29:45 +0100
Subject: [Catalog-sig] PEP 381 (Was: PyPI down?)
In-Reply-To: <4B54C8C8.6050703@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<4B54C8C8.6050703@v.loewis.de>
Message-ID: <84453B01-99A6-41F6-9E9D-E048D87418C5@leidel.info>

Am 18.01.2010 um 21:47 schrieb Martin v. L?wis:

>> FWIW, I don't see the PEP to be completed. The actual mirror
>> protocol, how to handle multiple (unsynchronized) indexes and the API
>> design are clearly undecided -- and not discussed.
> 
> The actual mirror protocol *is* decided, even though it's not explicitly
> spelled out in the PEP. It is based entirely on existing API; no new API
> on the PyPI side is planned. Basically, you do a lot of HTTP GETs.

If it's decided why isn't it public in the PEP? If you want developers to contribute you need stop deciding in private.

> As Tarek explains, the handling of unsynchronized indices is also
> discussed. Each mirror has a timestamp of last synchronization.
> 
> I expect that mirrors won't be behind more than two minutes. If you
> have a strong requirement to provide a better quality, you'll need
> to propose a PEP change. I would personally prefer to have such a
> feature only in a future revision of the protocol, based on practical
> experience.

See above.

> If you would like to propose a push model, I'd be curious what the
> advantage would be of one of the complicated-sounding APIs you mentioned
> over a simple trigger URL that the mirror might provide.

See mail to Tarek.

Jannis

From ziade.tarek at gmail.com  Mon Jan 18 22:44:10 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 18 Jan 2010 22:44:10 +0100
Subject: [Catalog-sig] PEP 381 (Was: PyPI down?)
In-Reply-To: <84453B01-99A6-41F6-9E9D-E048D87418C5@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<4B54C8C8.6050703@v.loewis.de>
	<84453B01-99A6-41F6-9E9D-E048D87418C5@leidel.info>
Message-ID: <94bdd2611001181344i1eab3e52g4c74eb557439524b@mail.gmail.com>

On Mon, Jan 18, 2010 at 10:29 PM, Jannis Leidel <jannis at leidel.info> wrote:
> Am 18.01.2010 um 21:47 schrieb Martin v. L?wis:
>
>>> FWIW, I don't see the PEP to be completed. The actual mirror
>>> protocol, how to handle multiple (unsynchronized) indexes and the API
>>> design are clearly undecided -- and not discussed.
>>
>> The actual mirror protocol *is* decided, even though it's not explicitly
>> spelled out in the PEP. It is based entirely on existing API; no new API
>> on the PyPI side is planned. Basically, you do a lot of HTTP GETs.
>
> If it's decided why isn't it public in the PEP? If you want developers to contribute you need stop deciding in private.

Come on, that's not what happened. IIRC Jim Fulton and Philip Eby
worked with Martin to make easy_install / zc.buildout calls on PyPI
efficient. Then I started the PEP later, but because I wanted to set
up an "official ring" of mirrors.

I am the one to blame because I didn't update that part of the PEP yet
consequently. But the PEP doesn't really address the protocol to be
used to browse PyPI. It's just informative. This is a de-facto
standard for years now (you use it everytime you install something
using pip or easy_install), and it was discussed in the Mailing Lists
back then when it was created. It didn't land in a PEP back then, like
other stuff don't.

So nothing was decided in private.

Now the push stuff could be great to have maybe, but I agree with
Martin that we should first setup the mirrors then learn from there.
(For the story, I've proposed a push stuff at first when we started to
discuss this PEP, but I eventually agreed that this was not the most
important thing to have our first version of a mirror ring).

Regards,
Tarek

From mal at egenix.com  Mon Jan 18 22:54:30 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 18 Jan 2010 22:54:30 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B1FE7B1.9030608@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>
	<4B1DB9BE.5040000@python.org> <4B1FE7B1.9030608@egenix.com>
Message-ID: <4B54D896.2060108@egenix.com>

Hi Steve,

has there been any progress on this ?

M.-A. Lemburg wrote:
> Steve Holden, Chairman, PSF wrote:
>> Adding a Google-like clause might make us seem less Draconian.
> 
> Here's a proposal for a less controversial text based on the Google
> terms:
> 
> """
> PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to
> PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following:
> 
> 1. Content is restricted to Python packages and related information only.
> 
> 2. Any content uploaded to PyPI is provided on a non-confidential basis.
> 
> 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce,
> distribute, transmit, display, perform, and publish the content, including in digital form. This
> licence is for the sole purpose of enabling the PSF to display, distribute and promote the content
> on PyPI.
> 
> 4. I represent and warrant that I have complied with all government regulations concerning the
> transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if
> I am subject to United States law, I represent and warrant that I have obtained the proper
> governmental authorization for the export of the content I upload. I further affirm that any content
> I provide is not intended for use by a government end-user as defined in part 772 of the United
> States Export Administration Regulations.
> """
> 
> The general terms on the python.org legal page would have to be
> changed in the same way.

I've attached a message explaining some of the reasons for part 4.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 18 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
-------------- next part --------------
An embedded message was scrubbed...
From: "M.-A. Lemburg" <mal at egenix.com>
Subject: Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI
	usage	agreement
Date: Fri, 11 Dec 2009 01:45:10 +0100
Size: 7173
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100118/325fbff5/attachment.eml>

From martin at v.loewis.de  Mon Jan 18 22:56:31 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 18 Jan 2010 22:56:31 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>
	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>
Message-ID: <4B54D90F.7020901@v.loewis.de>

> Besides the reference use case "mirroring"?
> 
> I don't know. Let's think about potential web services that could use information about a new software release:
> 
> Continous integration, yay!
> Social communities with voting and commenting, yay!
> Version control, yay!
> "Private" mirrors (as in "not advertised in the public")
> http://pypants.org
> http://heroku.com
> 
> and so on..

Can you please propose a specific change to something?

Regards,
Martin

From david.lyon at preisshare.net  Mon Jan 18 22:52:05 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 19 Jan 2010 08:52:05 +1100 (EST)
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>
	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>
	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>
	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>
	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>
Message-ID: <43298.115.128.33.171.1263851525.squirrel@syd-srv02.ezyreg.com>

> On Mon, Jan 18, 2010 at 7:29 PM, Jannis Leidel <jannis at leidel.info> wrote:
> [..]
>>> So in ?that case pip will just have to check for the last modified
>>> date, as explained in the PEP, to know how "fresh" a mirror is. The
>>> strategy to take depending on this freshness it's up to the client
>>> program.
>>
>> Really, pip gets to decide which package is fresh enough? Like a
>> PIP_BEST_BEFORE setting var?
>
> Pip gets to decide which *mirror* to use, given their last modified
> dates. I am not sure what would be the best strategy here.
>
>
>>>> Is this API going to be open for other non-official/non-open mirrors?
>>> Which API ?
>>
>> The API of PyPI which would actively ping mirrors to update their
>> package data.
>
> We v'e discussed this last year, and came up with the conclusion that
> asking PyPI to actively call each mirror was quite an intensive work
> because it means it has to call each mirror for each update (there are
> many updates per hour), and deal with a timeout for each request, etc.
> IOW, work *all day long* just for that.
>
> The other problem is that when a mirror is down or unreachable for a
> while, it can't get that ping. So what happens is that the mirror
> still needs to update itself in these situations. (because PyPI will
> certainly not implement a replay-system when some ping fails.)
>
> So why bother setting up two different update systems ? each mirror
> can look at the CHANGELOG every minute or so and get updated on their
> side.

You guys could make it a lot easier for yourselves and look
at twisted/jabber/rss.

They're industrial protocols for pushing out feed information
slowly. They take a few hours to set up and don't bog the
processors down in any way.

Jabber(twisted) and xmpp is very easy to push update information
messages out on.

David



From jannis at leidel.info  Mon Jan 18 23:06:58 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 23:06:58 +0100
Subject: [Catalog-sig] PEP 381 (Was: PyPI down?)
In-Reply-To: <94bdd2611001181344i1eab3e52g4c74eb557439524b@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B4B8C5D.3040409@v.loewis.de>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<4B54C8C8.6050703@v.loewis.de>
	<84453B01-99A6-41F6-9E9D-E048D87418C5@leidel.info>
	<94bdd2611001181344i1eab3e52g4c74eb557439524b@mail.gmail.com>
Message-ID: <FACB4FED-E226-4A4E-B698-9B60A4575BE9@leidel.info>


Am 18.01.2010 um 22:44 schrieb Tarek Ziad?:

> On Mon, Jan 18, 2010 at 10:29 PM, Jannis Leidel <jannis at leidel.info> wrote:
>> Am 18.01.2010 um 21:47 schrieb Martin v. L?wis:
>> 
>>>> FWIW, I don't see the PEP to be completed. The actual mirror
>>>> protocol, how to handle multiple (unsynchronized) indexes and the API
>>>> design are clearly undecided -- and not discussed.
>>> 
>>> The actual mirror protocol *is* decided, even though it's not explicitly
>>> spelled out in the PEP. It is based entirely on existing API; no new API
>>> on the PyPI side is planned. Basically, you do a lot of HTTP GETs.
>> 
>> If it's decided why isn't it public in the PEP? If you want developers to contribute you need stop deciding in private.
> 
> Come on, that's not what happened. IIRC Jim Fulton and Philip Eby
> worked with Martin to make easy_install / zc.buildout calls on PyPI
> efficient. Then I started the PEP later, but because I wanted to set
> up an "official ring" of mirrors.
> 
> I am the one to blame because I didn't update that part of the PEP yet
> consequently. But the PEP doesn't really address the protocol to be
> used to browse PyPI. It's just informative. This is a de-facto
> standard for years now (you use it everytime you install something
> using pip or easy_install), and it was discussed in the Mailing Lists
> back then when it was created. It didn't land in a PEP back then, like
> other stuff don't.
> 
> So nothing was decided in private.

It certainly feels like it if I get told that the API is decided.

Jannis

From jannis at leidel.info  Mon Jan 18 23:07:07 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 23:07:07 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B54D90F.7020901@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>
	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>
	<4B54D90F.7020901@v.loewis.de>
Message-ID: <9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>


Am 18.01.2010 um 22:56 schrieb Martin v. L?wis:

>> Besides the reference use case "mirroring"?
>> 
>> I don't know. Let's think about potential web services that could use information about a new software release:
>> 
>> Continous integration, yay!
>> Social communities with voting and commenting, yay!
>> Version control, yay!
>> "Private" mirrors (as in "not advertised in the public")
>> http://pypants.org
>> http://heroku.com
>> 
>> and so on..
> 
> Can you please propose a specific change to something?

I'm proposing:

Use push notifications (HTTP POST requests) to tell the mirrors about the availability of updates on the main main index, instead of having the mirrors pull that information periodically.

Jannis

From ziade.tarek at gmail.com  Mon Jan 18 23:14:45 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 18 Jan 2010 23:14:45 +0100
Subject: [Catalog-sig] PEP 381 (Was: PyPI down?)
In-Reply-To: <FACB4FED-E226-4A4E-B698-9B60A4575BE9@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<4B54C8C8.6050703@v.loewis.de>
	<84453B01-99A6-41F6-9E9D-E048D87418C5@leidel.info>
	<94bdd2611001181344i1eab3e52g4c74eb557439524b@mail.gmail.com>
	<FACB4FED-E226-4A4E-B698-9B60A4575BE9@leidel.info>
Message-ID: <94bdd2611001181414y4921175fh652d5fd8268334f3@mail.gmail.com>

On Mon, Jan 18, 2010 at 11:06 PM, Jannis Leidel <jannis at leidel.info> wrote:
[..]
>> So nothing was decided in private.
>
> It certainly feels like it if I get told that the API is decided.

I am not sure what you are putting behind the word "API". If you mean
the protocol to browse PyPI through HTTP requests, it was decided and
built in 2006/2007. Take a look at the archives.

Now for the mirroring stuff in PEP 381, comments/changes are welcome.



-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Mon Jan 18 23:16:42 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 18 Jan 2010 23:16:42 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <43298.115.128.33.171.1263851525.squirrel@syd-srv02.ezyreg.com>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B8CFF.9050505@simplistix.co.uk>	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>
	<43298.115.128.33.171.1263851525.squirrel@syd-srv02.ezyreg.com>
Message-ID: <4B54DDCA.2000109@v.loewis.de>

> You guys could make it a lot easier for yourselves and look
> at twisted/jabber/rss.

It would be actually much more difficult for me, unless you are willing
to completely design a detailed specification, and present it.

Regards,
Martin

From martin at v.loewis.de  Mon Jan 18 23:20:27 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 18 Jan 2010 23:20:27 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>
	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>
	<4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
Message-ID: <4B54DEAB.5010600@v.loewis.de>

>> Can you please propose a specific change to something?
> 
> I'm proposing:
> 
> Use push notifications (HTTP POST requests) to tell the mirrors about
> the availability of updates on the main main index, instead of having
> the mirrors pull that information periodically.

Ok: what specific notifications (i.e.: what specific URL, what specific
payload), and what specific servers to contact (i.e. where does the
list of servers come from)?

Regards,
Martin

From martin at v.loewis.de  Mon Jan 18 23:22:55 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 18 Jan 2010 23:22:55 +0100
Subject: [Catalog-sig] PEP 381 (Was: PyPI down?)
In-Reply-To: <FACB4FED-E226-4A4E-B698-9B60A4575BE9@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B8C5D.3040409@v.loewis.de>	<4B4B8CFF.9050505@simplistix.co.uk>	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<4B54C8C8.6050703@v.loewis.de>	<84453B01-99A6-41F6-9E9D-E048D87418C5@leidel.info>	<94bdd2611001181344i1eab3e52g4c74eb557439524b@mail.gmail.com>
	<FACB4FED-E226-4A4E-B698-9B60A4575BE9@leidel.info>
Message-ID: <4B54DF3F.5030408@v.loewis.de>

>> So nothing was decided in private.
> 
> It certainly feels like it if I get told that the API is decided.

It wasn't explicitly decided - I just assumed that the obvious
mechanisms would be used, so no real decision on anything was necessary.

In addition, it was not in private, but on this list - it's just not
reflected in the PEP.

Regards,
Martin

From jannis at leidel.info  Mon Jan 18 23:26:59 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Mon, 18 Jan 2010 23:26:59 +0100
Subject: [Catalog-sig] PEP 381 (Was: PyPI down?)
In-Reply-To: <94bdd2611001181414y4921175fh652d5fd8268334f3@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B4B8CFF.9050505@simplistix.co.uk>
	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>
	<4B4B9432.9050704@simplistix.co.uk>
	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>
	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>
	<4B54C8C8.6050703@v.loewis.de>
	<84453B01-99A6-41F6-9E9D-E048D87418C5@leidel.info>
	<94bdd2611001181344i1eab3e52g4c74eb557439524b@mail.gmail.com>
	<FACB4FED-E226-4A4E-B698-9B60A4575BE9@leidel.info>
	<94bdd2611001181414y4921175fh652d5fd8268334f3@mail.gmail.com>
Message-ID: <800A7662-8C2A-4940-BA83-B3AC951AE637@leidel.info>


Am 18.01.2010 um 23:14 schrieb Tarek Ziad?:

> On Mon, Jan 18, 2010 at 11:06 PM, Jannis Leidel <jannis at leidel.info> wrote:
> [..]
>>> So nothing was decided in private.
>> 
>> It certainly feels like it if I get told that the API is decided.
> 
> I am not sure what you are putting behind the word "API". If you mean
> the protocol to browse PyPI through HTTP requests, it was decided and
> built in 2006/2007. Take a look at the archives.

Sorry for the confusion, I was refering to Martin's comment:

"The actual mirror protocol *is* decided, even though it's not explicitly
spelled out in the PEP. It is based entirely on existing API; no new API
on the PyPI side is planned. Basically, you do a lot of HTTP GETs."

> Now for the mirroring stuff in PEP 381, comments/changes are welcome.

Great :)

Jannis

From robert.kern at gmail.com  Mon Jan 18 23:48:49 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 18 Jan 2010 16:48:49 -0600
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B54DEAB.5010600@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de>
Message-ID: <hj2ogi$kuo$1@ger.gmane.org>

On 2010-01-18 16:20 PM, "Martin v. L?wis" wrote:
>>> Can you please propose a specific change to something?
>>
>> I'm proposing:
>>
>> Use push notifications (HTTP POST requests) to tell the mirrors about
>> the availability of updates on the main main index, instead of having
>> the mirrors pull that information periodically.
>
> Ok: what specific notifications (i.e.: what specific URL, what specific
> payload), and what specific servers to contact (i.e. where does the
> list of servers come from)?

Not that I care too much[1], but one reasonable approach would be to use 
Google's PubSubHubbub protocol which attempts to standardize such push 
notifications.

   http://code.google.com/p/pubsubhubbub/

"""
The protocol in a nutshell is as follows:

* An feed URL (a "topic") declares its Hub server(s) in its Atom or RSS XML 
file, via <link rel="hub" ...>. The hub(s) can be run by the publisher of the 
feed, or can be a community hub that anybody can use. (Atom and RssFeeds are 
supported)

* A subscriber (a server that's interested in a topic), initially fetches the 
Atom URL as normal. If the Atom file declares its hubs, the subscriber can then 
avoid lame, repeated polling of the URL and can instead register with the feed's 
hub(s) and subscribe to updates.

* The subscriber subscribes to the Topic URL from the Topic URL's declared Hub(s).

* When the Publisher next updates the Topic URL, the publisher software pings 
the Hub(s) saying that there's an update.

* The hub efficiently fetches the published feed and multicasts the new/changed 
content out to all registered subscribers.
"""

[1] Personally, I probably won't use the API, so I have no dog in the fight, and 
I'm certainly not volunteering to code anything. I just wanted to add some more 
concreteness to what Jannis is suggesting.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ben+python at benfinney.id.au  Mon Jan 18 23:55:14 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 19 Jan 2010 09:55:14 +1100
Subject: [Catalog-sig] Setting up the Reply-To field in mailman
References: <94bdd2611001181011w4bb3a58dt7b47eafaf197da9a@mail.gmail.com>
Message-ID: <87pr57gh0t.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> Could we set the "Reply-To" field to the email of the mailing-list ?

Please, no munging of the reply-to field
<URL:http://woozle.org/~neale/papers/reply-to-still-harmful.html>.

> This is quite useful to avoid long CC lists and to avoid forgetting to
> CC the mailing-list

To avoid those problems, use (or, if it doesn't yet exist, lobby for)
the ?Reply to list? feature in your mail client. It will do the right
thing on a mailing list, thanks to the RFC 2369 fields in every message
from the list.

-- 
 \      ?I wish a robot would get elected president. That way, when he |
  `\    came to town, we could all take a shot at him and not feel too |
_o__)                                               bad.? ?Jack Handey |
Ben Finney


From david.lyon at preisshare.net  Mon Jan 18 23:53:57 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 18 Jan 2010 17:53:57 -0500
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B54DDCA.2000109@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B8CFF.9050505@simplistix.co.uk>	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>
	<43298.115.128.33.171.1263851525.squirrel@syd-srv02.ezyreg.com>
	<4B54DDCA.2000109@v.loewis.de>
Message-ID: <1f5a2daae4dccefc202fbcf1fcdae722@preisshare.net>

On Mon, 18 Jan 2010 23:16:42 +0100, "Martin v. L?wis" <martin at v.loewis.de>
wrote:
>> You guys could make it a lot easier for yourselves and look
>> at twisted/jabber/rss.
> 
> It would be actually much more difficult for me, unless you are willing
> to completely design a detailed specification, and present it.

Yes that's certainly possible. 

Let me just check my reading of the PEP-381. 

What you want is to allow external machines to get involved in the package 
delivery service. These machines all link up somehow and synchronise
somehow.

In addition to delivery of packages, they will collect and submit
statistics.

The current plan is via submission of an IP address.

If that is correct then it all sounds reasonable.

Only this week I came across a tiny twisted utility that was like
a mini jabberd.

What I am suggesting is that pypi run the jabberd or custom
jabberd. The mirrors connect to the jabberd.

The (external) mirrors are "up" when they are connected to
the jabberd and down when they are not.

Changes to packages can be fed instantly to all the mirror
clients (new packages) or delayed (a few seconds) if there is 
heavy load or too many clients.

If you choose standard jabberd there is some code overhead
and some extra learning. Balanced by maturity in the code
and protocol.

If you pick a custom jabberd like I saw in the example there
is more flexibility and it is easy to get started. The twisted
based server that I saw was under 50 lines of code.

Well, it's all very interresting stuff... :-)

PEP-381 looks like it is heading down the right track.

David








From jannis at leidel.info  Tue Jan 19 00:06:32 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Tue, 19 Jan 2010 00:06:32 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B54DEAB.5010600@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>
	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>
	<4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de>
Message-ID: <D38E9079-764E-40C1-B75B-2E9B0DBE0C68@leidel.info>

Am 18.01.2010 um 23:20 schrieb Martin v. L?wis:

>>> Can you please propose a specific change to something?
>> 
>> I'm proposing:
>> 
>> Use push notifications (HTTP POST requests) to tell the mirrors about
>> the availability of updates on the main main index, instead of having
>> the mirrors pull that information periodically.
> 
> Ok: what specific notifications (i.e.: what specific URL, what specific
> payload), and what specific servers to contact (i.e. where does the
> list of servers come from)?

Extra URLs of the mirror
------------------------

1. /update

receives a payload with a timestamp to tell the mirror to update itself
the mirror should use that timestamp to compare it to its own "last-modified" timestamp.

2. /update/<project-name>/<version>

receives metadata of each project (same metadata the XML-RPC API uses) as well as a timestamp.

Extra URLs of PyPI
------------------

None

In case a mirror doesn't accept the update payload (e.g. due to temporary loss of connectivity) the mirror needs to be able to catch up at a later time.

Correct me if I'm wrong, but I believe the XML-RPC API is capable of providing that information already. Once the mirror gained the list of projects still to be updated, it would mirror them "manually".

List of servers
---------------

Actually this wouldn't differ from the PEP: starting with trusted mirrors. And I absolutely agree with what you earlier said, to slowly start with such a new feature.

On the long run -- but I keep that for a later proposal -- there would be also a way to register non-mirror consumers of the push notifications on the PyPI website.

Jannis



From martin at v.loewis.de  Tue Jan 19 00:07:27 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 19 Jan 2010 00:07:27 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <hj2ogi$kuo$1@ger.gmane.org>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>	<4B54DEAB.5010600@v.loewis.de>
	<hj2ogi$kuo$1@ger.gmane.org>
Message-ID: <4B54E9AF.4020105@v.loewis.de>

> Not that I care too much[1], but one reasonable approach would be to use
> Google's PubSubHubbub protocol which attempts to standardize such push
> notifications.

Thanks. Now I would need two things:
1. a public statement by somebody that this is the protocol they would
   actually want to use. I still don't see the need for mirroring, so
   I would provide it independent of mirroring.
2. a recommendation what specific hub to use. I would prefer not to run
   my own, let alone implementing a hub as part of PyPI (although
   contributions are welcome).

I would then probably setup an RSS feed of the last hour of changes,
and arrange to ping the hub.

Regards,
Martin

From martin at v.loewis.de  Tue Jan 19 00:09:53 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 19 Jan 2010 00:09:53 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <1f5a2daae4dccefc202fbcf1fcdae722@preisshare.net>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B8CFF.9050505@simplistix.co.uk>	<94bdd2611001111307mfb9a59frf0feac93fdc18595@mail.gmail.com>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<43298.115.128.33.171.1263851525.squirrel@syd-srv02.ezyreg.com>	<4B54DDCA.2000109@v.loewis.de>
	<1f5a2daae4dccefc202fbcf1fcdae722@preisshare.net>
Message-ID: <4B54EA41.3090902@v.loewis.de>

> In addition to delivery of packages, they will collect and submit
> statistics.
> 
> The current plan is via submission of an IP address.
> 
> If that is correct then it all sounds reasonable.

I think Jannis' issue really isn't with the statistics, but with
updates to PyPI.

Regards,
Martin

From martin at v.loewis.de  Tue Jan 19 00:23:29 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 19 Jan 2010 00:23:29 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <D38E9079-764E-40C1-B75B-2E9B0DBE0C68@leidel.info>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>
	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>
	<4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de>
	<D38E9079-764E-40C1-B75B-2E9B0DBE0C68@leidel.info>
Message-ID: <4B54ED71.4000304@v.loewis.de>

> 1. /update
> 
> receives a payload with a timestamp to tell the mirror to update
> itself the mirror should use that timestamp to compare it to its own
> "last-modified" timestamp.
> 
> 2. /update/<project-name>/<version>
> 
> receives metadata of each project (same metadata the XML-RPC API
> uses) as well as a timestamp.

What's the purpose of the second URL? Isn't the first one sufficient?
And what is the version number for (what if there isn't really one, as
in project creation or deletion)?

Also, it would be possible to drop the timestamp from the first URL -
why have it?

> Correct me if I'm wrong, but I believe the XML-RPC API is capable of
> providing that information already. Once the mirror gained the list
> of projects still to be updated, it would mirror them "manually".

Exactly so, through the changelog call. It would actually be possible
to use that also in the /update URL.

> Actually this wouldn't differ from the PEP: starting with trusted
> mirrors. And I absolutely agree with what you earlier said, to slowly
> start with such a new feature.

Ok, but if there is no immediate need for such a feature outside of
mirroring, I'd really like to propose that this tight synchronization
for mirroring is unnecessary. Even with your proposed protocol, a
mirror might still be out of date (assuming PyPI notifies
asynchronously, which it IMO should). So applications that expect a
release to be available on all mirrors right after the POST to
PyPI returns would still find that they have to wait "somewhat".

Regards,
Martin


From david.lyon at preisshare.net  Tue Jan 19 00:22:56 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 18 Jan 2010 18:22:56 -0500
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B54E9AF.4020105@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>	<4B54DEAB.5010600@v.loewis.de>
	<hj2ogi$kuo$1@ger.gmane.org> <4B54E9AF.4020105@v.loewis.de>
Message-ID: <b80fa96f7edb5273c4ab98877eef635e@preisshare.net>

On Tue, 19 Jan 2010 00:07:27 +0100, "Martin v. L?wis" <martin at v.loewis.de>
wrote:

> Thanks. Now I would need two things:
> 1. a public statement by somebody that this is the protocol they would
>    actually want to use. I still don't see the need for mirroring, so
>    I would provide it independent of mirroring.

There are consumers for this sure.

Linux package maintainers, custom python distributions etc.

> 2. a recommendation what specific hub to use. I would prefer not to run
>    my own, let alone implementing a hub as part of PyPI (although
>    contributions are welcome).

Jabberd as an example is not overly complex to setup or administer.

You can control access with username/password pairs. Or allow self
registration.

> I would then probably setup an RSS feed of the last hour of changes,
> and arrange to ping the hub.

RSS are always open HTTP connections.

There is no need to ping. When there is new information it is
just written out on the socket.

Perphaps you could accomplish everything that you want with
RSS.

The "community" would be more impressed with a jabberd style
implementation as it can enable two-way communication.

Two-way communication could enable testbot and package testing
to take place. Spread around the community.

So in a way, two-way communication could really "open-up"
pypi and reduce the amount of time on your part that it
takes to maintain.

David



From martin at v.loewis.de  Tue Jan 19 00:42:25 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 19 Jan 2010 00:42:25 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>	<4B54DEAB.5010600@v.loewis.de>
	<hj2ogi$kuo$1@ger.gmane.org> <4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
Message-ID: <4B54F1E1.6050400@v.loewis.de>

David Lyon wrote:
> On Tue, 19 Jan 2010 00:07:27 +0100, "Martin v. L?wis" <martin at v.loewis.de>
> wrote:
> 
>> Thanks. Now I would need two things:
>> 1. a public statement by somebody that this is the protocol they would
>>    actually want to use. I still don't see the need for mirroring, so
>>    I would provide it independent of mirroring.
> 
> There are consumers for this sure.
> 
> Linux package maintainers, custom python distributions etc.

Which one specifically?

>> 2. a recommendation what specific hub to use. I would prefer not to run
>>    my own, let alone implementing a hub as part of PyPI (although
>>    contributions are welcome).
> 
> Jabberd as an example is not overly complex to setup or administer.

Hmm. AFAICT, jabberd does not support pubsubhubbub.

> 
> You can control access with username/password pairs. Or allow self
> registration.

See, this is precisely the kind of stuff I don't have *any* cycles for.
I don't want to run another service, and worry about people using it
to break into python.org.

>> I would then probably setup an RSS feed of the last hour of changes,
>> and arrange to ping the hub.
> 
> RSS are always open HTTP connections.

Sorry, but that's how pubsubhubbhub works.

> Perphaps you could accomplish everything that you want with
> RSS.

I didn't propose RSS. Robert Kern did, by proposing to use
pubsubhubbub (although the specification seems to favor ATOM).

> The "community" would be more impressed with a jabberd style
> implementation as it can enable two-way communication.

The "community" just proposed the contrary (more precisely,
three different people proposing three different technologies).
Please don't speak for anybody but yourself (unless you are
*certain* that you have been authorized to do so).

Regards,
Martin

From david.lyon at preisshare.net  Tue Jan 19 00:53:41 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 18 Jan 2010 18:53:41 -0500
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B54F1E1.6050400@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>	<4B54DEAB.5010600@v.loewis.de>
	<hj2ogi$kuo$1@ger.gmane.org> <4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
Message-ID: <a550e45c26df80bbebea6492327d5fbf@preisshare.net>

On Tue, 19 Jan 2010 00:42:25 +0100, "Martin v. L?wis" <martin at v.loewis.de>
wrote:

> The "community" just proposed the contrary (more precisely,
> three different people proposing three different technologies).
> Please don't speak for anybody but yourself (unless you are
> *certain* that you have been authorized to do so).

No need to get het up. 

We're just having a friendly discussion Martin.  You asked
a question and some people answered it.

Three different solutions is healthy discussion.

It would be a big surprise to everybody if the architecture
or infrastructure got "opened" up. That's what Jannis was
talking about.

You're right.. everything is tightly controlled.. and
every small step needs "authorization". That would never
happen with CPAN. It's why we're not going that way..

Take care

David


From david.lyon at preisshare.net  Tue Jan 19 01:42:37 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 18 Jan 2010 19:42:37 -0500
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B54F1E1.6050400@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>	<4B54DEAB.5010600@v.loewis.de>
	<hj2ogi$kuo$1@ger.gmane.org> <4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
Message-ID: <951a972618365a1aafdad0615895f0e9@preisshare.net>

On Tue, 19 Jan 2010 00:42:25 +0100, "Martin v. L?wis" <martin at v.loewis.de>
wrote:

>>> 1. a public statement by somebody that this is the protocol they would
>>>    actually want to use. I still don't see the need for mirroring, so
>>>    I would provide it independent of mirroring.
>> 
>> There are consumers for this sure.
>> 
>> Linux package maintainers, custom python distributions etc.
> 
> Which one specifically?

Linux package maintainers could find a use for it because they
could more quickly get updates from package maintainers. Specifically,
any distro that repackages from pypi.

Custom Distributions : ActiveState

Also, end users. Even end users could read a package feed from
pypi and it could enable an update. That would be good for python 3.


>>> 2. a recommendation what specific hub to use. I would prefer not to run
>>>    my own, let alone implementing a hub as part of PyPI (although
>>>    contributions are welcome).

You don't need to run it. It can be run "anywhere" on the internet.

>> You can control access with username/password pairs. Or allow self
>> registration.
> 
> See, this is precisely the kind of stuff I don't have *any* cycles for.
> I don't want to run another service, and worry about people using it
> to break into python.org.

Oh please. A jabber server can run on any machine 'anywhere' on the
internet.

Saying that running a jabber service on some machine will bring down 
python.org stretches my imagination at least.

> I didn't propose RSS. Robert Kern did, by proposing to use
> pubsubhubbub (although the specification seems to favor ATOM).

Well it's probably a good suggestion. I'm sure you will come
to some decision.

David


From martin at v.loewis.de  Tue Jan 19 03:24:13 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 19 Jan 2010 03:24:13 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <951a972618365a1aafdad0615895f0e9@preisshare.net>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>	<4B54DEAB.5010600@v.loewis.de>
	<hj2ogi$kuo$1@ger.gmane.org> <4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
Message-ID: <4B5517CD.2050406@v.loewis.de>

>>>> 1. a public statement by somebody that this is the protocol they would
>>>>    actually want to use. I still don't see the need for mirroring, so
>>>>    I would provide it independent of mirroring.
>>> There are consumers for this sure.
>>>
>>> Linux package maintainers, custom python distributions etc.
>> Which one specifically?
> 
> Linux package maintainers could find a use for it because they
> could more quickly get updates from package maintainers. Specifically,
> any distro that repackages from pypi.
> 
> Custom Distributions : ActiveState
> 
> Also, end users. Even end users could read a package feed from
> pypi and it could enable an update. That would be good for python 3.

I'm giving up. Who, inside ActiveState, have you talked to who said
that ActiveState wants to use pubsubhubbub for receiving change
notifications from PyPI? I suppose the answer is "Nobody" - you
are just *assuming* that ActiveState may want to use pubsubhubbub.

Regards,
Martin

From justin at justinlilly.com  Tue Jan 19 03:35:03 2010
From: justin at justinlilly.com (Justin Lilly)
Date: Mon, 18 Jan 2010 21:35:03 -0500
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B5517CD.2050406@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk> <4B54D90F.7020901@v.loewis.de> 
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de> 
	<hj2ogi$kuo$1@ger.gmane.org> <4B54E9AF.4020105@v.loewis.de> 
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de> 
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
Message-ID: <b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>

On Mon, Jan 18, 2010 at 9:24 PM, "Martin v. L?wis" <martin at v.loewis.de>wrote:

> >>>> 1. a public statement by somebody that this is the protocol they would
> >>>>    actually want to use. I still don't see the need for mirroring, so
> >>>>    I would provide it independent of mirroring.
> >>> There are consumers for this sure.
> >>>
> >>> Linux package maintainers, custom python distributions etc.
> >> Which one specifically?
> >
> > Linux package maintainers could find a use for it because they
> > could more quickly get updates from package maintainers. Specifically,
> > any distro that repackages from pypi.
> >
> > Custom Distributions : ActiveState
> >
> > Also, end users. Even end users could read a package feed from
> > pypi and it could enable an update. That would be good for python 3.
>
> I'm giving up. Who, inside ActiveState, have you talked to who said
> that ActiveState wants to use pubsubhubbub for receiving change
> notifications from PyPI? I suppose the answer is "Nobody" - you
> are just *assuming* that ActiveState may want to use pubsubhubbub.


If I might, I, as a community member, would like to use pubsubhubbub. I've
had 2 occasions on projects where it would have proven useful, both needed
to be notified of updates to specific packages, but had nothing directly to
do with downloading and installing packages.

What does that count for?

 -justin



>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100118/b6fa7734/attachment.htm>

From david.lyon at preisshare.net  Tue Jan 19 03:30:11 2010
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 18 Jan 2010 21:30:11 -0500
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B5517CD.2050406@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>	<4B4B9432.9050704@simplistix.co.uk>	<94bdd2611001111322g253e6991i2552c7cbef6a0f24@mail.gmail.com>	<420ACB91-FCCC-48B7-912E-D55D45C8CC57@leidel.info>	<94bdd2611001180925v7d16fcd5o5b9e6a19aa85650c@mail.gmail.com>	<0E427E5C-80E2-4DB2-80D0-13A6184DD194@leidel.info>	<94bdd2611001181003y237140ecl99b47eaf9293496f@mail.gmail.com>	<BE5B7CD2-ABDD-4C88-B34C-BAB8DE04A4DC@leidel.info>	<94bdd2611001181049t69aec855vb6ec1132db0447f4@mail.gmail.com>	<79B8C09E-18D3-4409-88EE-F27F181BBA1D@leidel.info>	<94bdd2611001181140s5af2a6e3t4209b57fa314d3dd@mail.gmail.com>	<C60A906F-F35A-4EF0-AD00-5C177368F18D@leidel.info>	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>	<4B54DEAB.5010600@v.loewis.de>
	<hj2ogi$kuo$1@ger.gmane.org> <4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
Message-ID: <76f710cc29073757b0ae6d2ec759bbce@preisshare.net>

On Tue, 19 Jan 2010 03:24:13 +0100, "Martin v. L?wis" <martin at v.loewis.de>
wrote:
>>>>> 1. a public statement by somebody that this is the protocol they
would
>>>>>    actually want to use. I still don't see the need for mirroring, so
>>>>>    I would provide it independent of mirroring.
>>>> There are consumers for this sure.
>>>>
>>>> Linux package maintainers, custom python distributions etc.
>>> Which one specifically?
>> 
>> Linux package maintainers could find a use for it because they
>> could more quickly get updates from package maintainers. Specifically,
>> any distro that repackages from pypi.
>> 
>> Custom Distributions : ActiveState
>> 
>> Also, end users. Even end users could read a package feed from
>> pypi and it could enable an update. That would be good for python 3.
> 
> I'm giving up. Who, inside ActiveState, have you talked to who said
> that ActiveState wants to use pubsubhubbub for receiving change
> notifications from PyPI? I suppose the answer is "Nobody" - you
> are just *assuming* that ActiveState may want to use pubsubhubbub.

That wasn't the question the question you asked.

Can we get back to talking about mirroring..?

Regards

David


From martin at v.loewis.de  Tue Jan 19 03:48:30 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 19 Jan 2010 03:48:30 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de> <hj2ogi$kuo$1@ger.gmane.org>
	<4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
	<b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
Message-ID: <4B551D7E.50806@v.loewis.de>

> If I might, I, as a community member, would like to use pubsubhubbub.
> I've had 2 occasions on projects where it would have proven useful, both
> needed to be notified of updates to specific packages, but had nothing
> directly to do with downloading and installing packages.
> 
> What does that count for?

It's a starting point - are you also willing to test some infrastructure
that I would set up (say, within the next week or so)?

Then coming to the second question: which hub should I use?

Regards,
Martin

From ssteinerx at gmail.com  Tue Jan 19 13:15:09 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 19 Jan 2010 07:15:09 -0500
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B551D7E.50806@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk> <4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de> <hj2ogi$kuo$1@ger.gmane.org>
	<4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
	<b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
	<4B551D7E.50806@v.loewis.de>
Message-ID: <A715840A-21D5-4988-9736-2B46FB622437@gmail.com>


On Jan 18, 2010, at 9:48 PM, Martin v. L?wis wrote:

>> If I might, I, as a community member, would like to use pubsubhubbub.
>> I've had 2 occasions on projects where it would have proven useful, both
>> needed to be notified of updates to specific packages, but had nothing
>> directly to do with downloading and installing packages.
>> 
>> What does that count for?
> 
> It's a starting point - are you also willing to test some infrastructure
> that I would set up (say, within the next week or so)?

I can test it on the receiving end; I'm working on packaging up my pymetamirror package and putting the results on S3.  

It'd be cool to have it get notifications of what to download instead of doing a date based pull as is the current setup.

S


From steve at holdenweb.com  Tue Jan 19 05:21:58 2010
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 18 Jan 2010 23:21:58 -0500
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B54D896.2060108@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>
	<4B1DB9BE.5040000@python.org> <4B1FE7B1.9030608@egenix.com>
	<4B54D896.2060108@egenix.com>
Message-ID: <4B553366.2090809@holdenweb.com>

None that I am aware of, but Martin is the one who's been making changes
most recently. I don't think there's been any input from Van on this
yet, but I've been busy and may have forgotten or missed something.

regards
 Steve

M.-A. Lemburg wrote:
> Hi Steve,
> 
> has there been any progress on this ?
> 
> M.-A. Lemburg wrote:
>> Steve Holden, Chairman, PSF wrote:
>>> Adding a Google-like clause might make us seem less Draconian.
>> Here's a proposal for a less controversial text based on the Google
>> terms:
>>
>> """
>> PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to
>> PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following:
>>
>> 1. Content is restricted to Python packages and related information only.
>>
>> 2. Any content uploaded to PyPI is provided on a non-confidential basis.
>>
>> 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce,
>> distribute, transmit, display, perform, and publish the content, including in digital form. This
>> licence is for the sole purpose of enabling the PSF to display, distribute and promote the content
>> on PyPI.
>>
>> 4. I represent and warrant that I have complied with all government regulations concerning the
>> transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if
>> I am subject to United States law, I represent and warrant that I have obtained the proper
>> governmental authorization for the export of the content I upload. I further affirm that any content
>> I provide is not intended for use by a government end-user as defined in part 772 of the United
>> States Export Administration Regulations.
>> """
>>
>> The general terms on the python.org legal page would have to be
>> changed in the same way.
> 
> I've attached a message explaining some of the reasons for part 4.
> 
> Thanks,
> 
> 
> ------------------------------------------------------------------------
> 
> Subject:
> Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
> From:
> "M.-A. Lemburg" <mal at egenix.com>
> Date:
> Fri, 11 Dec 2009 01:45:10 +0100
> To:
> Terry Reedy <tjreedy at udel.edu>
> 
> To:
> Terry Reedy <tjreedy at udel.edu>
> CC:
> catalog-sig at python.org, VanL <van at python.org>
> 
> 
> Terry Reedy wrote:
>> M.-A. Lemburg wrote:
>>> Steve Holden, Chairman, PSF wrote:
>>>> Adding a Google-like clause might make us seem less Draconian.
>>> Here's a proposal for a less controversial text based on the Google
>>> terms:
>> I like the third part better.
> 
> Thanks.
> 
>>> """
>>> PyPI is a service provided by the PSF. In order to be able to
>>> distribute the content you upload to
>>> PyPI to web site users, the PSF asks you to agree to and affirmatively
>>> acknowledge the following:
>>>
>>> 1. Content is restricted to Python packages and related information only.
>>>
>>> 2. Any content uploaded to PyPI is provided on a non-confidential basis.
>>>
>>> 3. The PSF is granted an irrevocable, worldwide, royalty-free,
>>> nonexclusive license to reproduce,
>>> distribute, transmit, display, perform, and publish the content,
>>> including in digital form. This
>>> licence is for the sole purpose of enabling the PSF to display,
>>> distribute and promote the content
>>> on PyPI.
>>>
>>> 4. I represent and warrant that I have complied with all government
>>> regulations concerning the
>>> transfer or export of any content I upload to the PyPI servers in The
>>> Netherlands. In particular, if
>>> I am subject to United States law, I represent and warrant that I have
>>> obtained the proper
>>> governmental authorization for the export of the content I upload. I
>>> further affirm that any content
>>> I provide is not intended for use by a government end-user as defined
>>> in part 772 of the United
>>> States Export Administration Regulations.
>>> """
>> The fourth section might scare people off without further explanation
>> somewhere, as it could be taken to imply that people have to get a US
>> gov permit to upload, which almost no one has done. If this is only
>> about crypto software, it should say so. I do not understand the last
>> sentence at all as open-source licenses do not usually exclude specific
>> users. I cannot affirm something that is complete gobble talk to me.
> 
> The clause has three parts:
> 
>  a) "I represent and warrant that I have complied with all government regulations concerning the
> transfer or export of any content I upload to the PyPI servers in The Netherlands."
> 
> This part is written in a general way and is needed to
> cover export regulations which may be imposed by the country
> of the uploader when uploading (exporting) applications to
> a server in the The Netherlands.
> 
> For many countries these export regulations are variants
> of the things laid out in the Wassenaar Arrangement which
> covers crypto code, but also other software technologies
> that may be considered dual-use:
> 
> http://www.wassenaar.org/
> in particular:
> http://www.wassenaar.org/controllists/2009/WA-LIST%20%2809%29%201/WA-LIST%20%2809%29%201.pdf
> 
> Most software will fall under the "GENERAL SOFTWARE NOTE"
> (with some special rules for crypto software), but countries
> may still implement additional rules such as the ones currently
> imposed by the US (you have to send them an email with the link
> to the download location - see
> http://www.bis.doc.gov/encryption/pubavailencsourcecodenofify.html).
> 
> Since the exact regulations depend on the country from where
> the code is uploaded, the clause can't be more specific.
> 
> I added the location of the servers to the original clause to
> make the export nature of the upload more specific.
> 
>  b) "In particular, if I am subject to United States law, I represent and warrant that I have
> obtained the proper governmental authorization for the export of the content I upload."
> 
> This part only applies to US uploaders.
> 
> Note that the US regulations have a subtle detail: they apply to
> all US-origin content. E.g. if you export some dual-use system software
> written in the US from Germany to Cuba, the US can put you on their
> embargo list.
> 
>  c)  "I further affirm that any content I provide is not intended for use by a government end-user
> as defined in part 772 of the United States Export Administration Regulations."
> 
> This part applies to all uploaders. The restriction appears to be
> a super-set of the embargo restrictions for various individuals -
> most of those are government end-users.
> 
> I find that clause too board as well, since it prevents government
> users in general to use PyPI packages.
> 
> Furthermore, the embargo lists also includes companies and, of course,
> whole countries, which this clause does not cover. See e.g.
> EU: http://ec.europa.eu/external_relations/cfsp/sanctions/docs/measures_en.pdf
> US: http://www.bis.doc.gov/news/2009/2009-fpr.pdf
> (note how e.g. Cuba is on the US list, but not on the EU list)
> 
> I'm not sure why the clause is needed. Perhaps Van could clarify
> this.
> 
> IMHO, part a) already covers everything that is needed w/r to
> export restrictions.
> 
> All this with the usual IANAL disclaimer. I've read a lot on these
> things when we started shipping a pyOpenSSL distribution. Some of the
> things I found are listed above.
> 


-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010  http://us.pycon.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/

From ronaldoussoren at mac.com  Tue Jan 19 13:51:37 2010
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 19 Jan 2010 13:51:37 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <A715840A-21D5-4988-9736-2B46FB622437@gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de> <hj2ogi$kuo$1@ger.gmane.org>
	<4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
	<b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
	<4B551D7E.50806@v.loewis.de>
	<A715840A-21D5-4988-9736-2B46FB622437@gmail.com>
Message-ID: <4E3A3009-BB5A-4E13-BA83-46E9AD2B31CB@mac.com>


On 19 Jan, 2010, at 13:15, ssteinerX at gmail.com wrote:

> 
> On Jan 18, 2010, at 9:48 PM, Martin v. L?wis wrote:
> 
>>> If I might, I, as a community member, would like to use pubsubhubbub.
>>> I've had 2 occasions on projects where it would have proven useful, both
>>> needed to be notified of updates to specific packages, but had nothing
>>> directly to do with downloading and installing packages.
>>> 
>>> What does that count for?
>> 
>> It's a starting point - are you also willing to test some infrastructure
>> that I would set up (say, within the next week or so)?
> 
> I can test it on the receiving end; I'm working on packaging up my pymetamirror package and putting the results on S3.  
> 
> It'd be cool to have it get notifications of what to download instead of doing a date based pull as is the current setup.

The date-based pulling would IMO be more robust when there can be intermittent connection problems (when your mirror is temporarily down, or there is some network problem).  

BTW. the payload for the proposed '/update' URL on mirrors isn't strictly necessary, but could be treated as a hint to perform an update in the near future.

That said, pushing updates to mirrors is probably overkill, CPAN's FAQ says that updating the mirror once a day is good enough (<http://www.cpan.org/misc/cpan-faq.html#How_mirror_CPAN>).

Ronald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3567 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100119/da4a1775/attachment.bin>

From robert.kern at gmail.com  Tue Jan 19 17:49:09 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 19 Jan 2010 10:49:09 -0600
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B551D7E.50806@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk>
	<4B54D90F.7020901@v.loewis.de>	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>	<4B54DEAB.5010600@v.loewis.de>
	<hj2ogi$kuo$1@ger.gmane.org>	<4B54E9AF.4020105@v.loewis.de>	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>	<4B54F1E1.6050400@v.loewis.de>	<951a972618365a1aafdad0615895f0e9@preisshare.net>	<4B5517CD.2050406@v.loewis.de>	<b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
	<4B551D7E.50806@v.loewis.de>
Message-ID: <hj4nq4$cna$1@ger.gmane.org>

On 2010-01-18 20:48 PM, "Martin v. L?wis" wrote:

> Then coming to the second question: which hub should I use?

I would recommend http://pubsubhubbub.appspot.com/. I don't think it will ever 
go away until well after Google eventually abandons the PubSubHubbub protocol 
for something else. And probably not even then.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ianb at colorstudy.com  Tue Jan 19 21:08:23 2010
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 19 Jan 2010 14:08:23 -0600
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <4B54CB54.9020001@v.loewis.de>
References: <4B54CB54.9020001@v.loewis.de>
Message-ID: <b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>

On Mon, Jan 18, 2010 at 2:57 PM, "Martin v. L?wis" <martin at v.loewis.de>wrote:

> I would like to change the simple API in the following
> ways, within a few days:
>
> 1. XML-escape href attributes to make the pages well-formed XML
>   (possibly other changes required for that as well, but I only
>    ran into lack of &amp;)
> 2. Make file paths relative (from http://pypi.python.org/packages
>   to ../../packages)
>
> Objections?
>

This should be tested with easy_install/pip.  I'm pretty sure 2 will be fine
but 1 could break things if those clients aren't doing unescaping.  I'm
pretty sure pip is not unescaping.

-- 
Ian Bicking  |  http://blog.ianbicking.org  |
http://topplabs.org/civichacker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100119/bf7681f3/attachment.htm>

From martin at v.loewis.de  Tue Jan 19 21:44:22 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 19 Jan 2010 21:44:22 +0100
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
Message-ID: <4B5619A6.4050701@v.loewis.de>

>     Objections?
> 
> 
> This should be tested with easy_install/pip.  I'm pretty sure 2 will be
> fine but 1 could break things if those clients aren't doing unescaping.
>  I'm pretty sure pip is not unescaping.

So how could this test be performed?

Regards,
Martin

From pje at telecommunity.com  Tue Jan 19 22:05:43 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 19 Jan 2010 16:05:43 -0500
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.co
 m>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
Message-ID: <20100119210547.08AEE3A4065@sparrow.telecommunity.com>

At 02:08 PM 1/19/2010 -0600, Ian Bicking wrote:
>On Mon, Jan 18, 2010 at 2:57 PM, "Martin v. L??wis" 
><<mailto:martin at v.loewis.de>martin at v.loewis.de> wrote:
>I would like to change the simple API in the following
>ways, within a few days:
>
>1. XML-escape href attributes to make the pages well-formed XML
>?  (possibly other changes required for that as well, but I only
>?  ? ran into lack of &amp;)
>2. Make file paths relative (from 
><http://pypi.python.org/packages>http://pypi.python.org/packages
>?  to ../../packages)
>
>Objections?
>
>
>This should be tested with easy_install/pip. ? I'm pretty sure 2 
>will be fine but 1 could break things if those clients aren't doing 
>unescaping. ? I'm pretty sure pip is not unescaping.

If you're using setuptools to do your PyPI access in pip, you are 
indeed unescaping.  ;-)  setuptools.package_index does full HTML 
entity decoding, including both named and numeric references, so XML 
escaping (which is a subset of that) should be just fine.

Likewise, #2 should be fine as long as urlparse.urljoin produces the 
correct result using the retrieved-from URL as the base URL.  I 
stress this because some web applications treat the urls "foo" and 
"foo/" as if they were interchangeable, and for relative-URL 
purposes, they are not.  So, as long as PyPI handles this correctly 
based on the URL that was sent by the *client* (rather than perhaps a 
canonical URL the client "should have" sent), then that will be fine. 


From martin at v.loewis.de  Tue Jan 19 22:21:25 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 19 Jan 2010 22:21:25 +0100
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <20100119210547.08AEE3A4065@sparrow.telecommunity.com>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
	<20100119210547.08AEE3A4065@sparrow.telecommunity.com>
Message-ID: <4B562255.2050709@v.loewis.de>

>> 2. Make file paths relative (from
>> <http://pypi.python.org/packages>http://pypi.python.org/packages
>> ?  to ../../packages)
...
> Likewise, #2 should be fine as long as urlparse.urljoin produces the
> correct result using the retrieved-from URL as the base URL.  I stress
> this because some web applications treat the urls "foo" and "foo/" as if
> they were interchangeable, and for relative-URL purposes, they are not. 
> So, as long as PyPI handles this correctly based on the URL that was
> sent by the *client* (rather than perhaps a canonical URL the client
> "should have" sent), then that will be fine.

Can you please suggest an example where this could break? If the client
doesn't send an absolute path, it will be incorrect HTTP, no?

Regards,
Martin


From martin at v.loewis.de  Wed Jan 20 01:11:06 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 20 Jan 2010 01:11:06 +0100
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <A715840A-21D5-4988-9736-2B46FB622437@gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de> <hj2ogi$kuo$1@ger.gmane.org>
	<4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
	<b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
	<4B551D7E.50806@v.loewis.de>
	<A715840A-21D5-4988-9736-2B46FB622437@gmail.com>
Message-ID: <4B564A1A.3030402@v.loewis.de>

> I can test it on the receiving end; I'm working on packaging up my pymetamirror package and putting the results on S3.  

Ok, I have now created

http://pypi.python.org/pypi?:action=lasthour

and registered it with

http://pubsubhubbub.appspot.com/publish

Please let me know whether you manage to receive notifications.

Regards,
Martin

From david.lyon at pythontest.org  Wed Jan 20 01:23:21 2010
From: david.lyon at pythontest.org (David Lyon)
Date: Wed, 20 Jan 2010 11:23:21 +1100 (EST)
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <4B564A1A.3030402@v.loewis.de>
References: <4B4B80E2.60804@simplistix.co.uk> <4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de> <hj2ogi$kuo$1@ger.gmane.org>
	<4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
	<b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
	<4B551D7E.50806@v.loewis.de>
	<A715840A-21D5-4988-9736-2B46FB622437@gmail.com>
	<4B564A1A.3030402@v.loewis.de>
Message-ID: <1475.218.214.45.58.1263947001.squirrel@syd-srv02.ezyreg.com>

Martin wrote:

> Ok, I have now created
>
> http://pypi.python.org/pypi?:action=lasthour
>
> and registered it with
>
> http://pubsubhubbub.appspot.com/publish
>
> Please let me know whether you manage to receive notifications.

Works here too..

That will save me an hour each day.. at least..

David







From ssteinerx at gmail.com  Wed Jan 20 01:44:12 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 19 Jan 2010 19:44:12 -0500
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <1475.218.214.45.58.1263947001.squirrel@syd-srv02.ezyreg.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de> <hj2ogi$kuo$1@ger.gmane.org>
	<4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
	<b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
	<4B551D7E.50806@v.loewis.de>
	<A715840A-21D5-4988-9736-2B46FB622437@gmail.com>
	<4B564A1A.3030402@v.loewis.de>
	<1475.218.214.45.58.1263947001.squirrel@syd-srv02.ezyreg.com>
Message-ID: <1BCB522D-D1E6-46FD-9540-D4E17CF80673@gmail.com>


On Jan 19, 2010, at 7:23 PM, David Lyon wrote:

> Martin wrote:
> 
>> Ok, I have now created
>> 
>> http://pypi.python.org/pypi?:action=lasthour
>> 
>> and registered it with
>> 
>> http://pubsubhubbub.appspot.com/publish
>> 
>> Please let me know whether you manage to receive notifications.
> 
> Works here too..
> 
> That will save me an hour each day.. at least..

David, if your code's checked in, you could save me an hour right now!

S


From pje at telecommunity.com  Wed Jan 20 01:57:50 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 19 Jan 2010 19:57:50 -0500
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <4B562255.2050709@v.loewis.de>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
	<20100119210547.08AEE3A4065@sparrow.telecommunity.com>
	<4B562255.2050709@v.loewis.de>
Message-ID: <20100120005756.8D9B13A4065@sparrow.telecommunity.com>

At 10:21 PM 1/19/2010 +0100, Martin v. L?wis wrote:
> >> 2. Make file paths relative (from
> >> <http://pypi.python.org/packages>http://pypi.python.org/packages
> >> ?  to ../../packages)
>...
> > Likewise, #2 should be fine as long as urlparse.urljoin produces the
> > correct result using the retrieved-from URL as the base URL.  I stress
> > this because some web applications treat the urls "foo" and "foo/" as if
> > they were interchangeable, and for relative-URL purposes, they are not.
> > So, as long as PyPI handles this correctly based on the URL that was
> > sent by the *client* (rather than perhaps a canonical URL the client
> > "should have" sent), then that will be fine.
>
>Can you please suggest an example where this could break? If the client
>doesn't send an absolute path, it will be incorrect HTTP, no?

No, what I mean is that if PyPI returns *identical* HTML for requests 
going to "/foo" and "/foo/", then one of those pages will contain 
incorrect relative URLs.

I mention this because, at least in the main web interface, going to 
http://pypi.python.org/pypi/setuptools and 
http://pypi.python.org/pypi/setuptools/ produces similar (if not 
identical) HTML.

Currently, the relative URLs in such pages are site-absolute, i.e., 
"/pypi/whatever", so the fact that the two pages are identical isn't 
a problem.  However, if you used pure relative URLs ('../whatever'), 
then one of the two pages would have incorrect relative URLs.

So, all I am saying is, please make sure that the generated relative 
URLs are correct, based not on a canonical form of the URL, but based 
on whether the client asked for 
http://pypi.python.org/simple/setuptools or 
http://pypi.python.org/simple/setuptools/ - because if those two URLs 
return *identical* HTML, *one* of them will be wrong.

In other words, one of those two pages must have one more '../' in 
each URL than the other one has for the same URL.  Does that make sense now?

I'm really just saying that the relative URLs must be correct, and 
reminding you (in case you're not already aware) that the current 
PyPI code may be relying on its use of "/pypi" relative URLs in order 
to ignore the trailing-slash issue.  And if that's the case, then 
there will be pages generated with *wrong* relative URLs, unless you 
remember to test both the '/'-terminated and non-'/'-terminated 
versions.  (Even if just by clicking the links in a browser.)


From david.lyon at pythontest.org  Wed Jan 20 01:58:50 2010
From: david.lyon at pythontest.org (David Lyon)
Date: Wed, 20 Jan 2010 11:58:50 +1100 (EST)
Subject: [Catalog-sig] PyPI and PEP 381
In-Reply-To: <1BCB522D-D1E6-46FD-9540-D4E17CF80673@gmail.com>
References: <4B4B80E2.60804@simplistix.co.uk> <4B54D90F.7020901@v.loewis.de>
	<9E8EAF43-DFF2-4FB3-A1B4-D23547942C71@leidel.info>
	<4B54DEAB.5010600@v.loewis.de> <hj2ogi$kuo$1@ger.gmane.org>
	<4B54E9AF.4020105@v.loewis.de>
	<b80fa96f7edb5273c4ab98877eef635e@preisshare.net>
	<4B54F1E1.6050400@v.loewis.de>
	<951a972618365a1aafdad0615895f0e9@preisshare.net>
	<4B5517CD.2050406@v.loewis.de>
	<b3cdbdcf1001181835q6b71ed68vbe05ad84036c4e04@mail.gmail.com>
	<4B551D7E.50806@v.loewis.de>
	<A715840A-21D5-4988-9736-2B46FB622437@gmail.com>
	<4B564A1A.3030402@v.loewis.de>
	<1475.218.214.45.58.1263947001.squirrel@syd-srv02.ezyreg.com>
	<1BCB522D-D1E6-46FD-9540-D4E17CF80673@gmail.com>
Message-ID: <1511.218.214.45.58.1263949130.squirrel@syd-srv02.ezyreg.com>

> On Jan 19, 2010, at 7:23 PM, Steve wrote:
>
> David, if your code's checked in, you could save me an hour right now!

I only got the xml a little while ago.. I haven't had a chance to
decode it yet. That will take a little more time.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rss PUBLIC "-//Netscape Communications//DTD RSS 0.91//EN"
"http://my.netscape.com/publish/formats/rss-0.91.dtd">
<rss version="0.91">
 <channel>
  <title>PyPI changes</title>
  <link>http://pypi.python.org/pypi</link>
  <description>Updates to the Python Package Index in the last
hour</description>
  <language>en</language>
  <item>
    <title>defensio</title>
    <link>http://pypi.python.org/pypi/defensio/0.9.1</link>
    <description>add source file defensio-0.9.1.tar.gz</description>
    <pubDate>20 Jan 2010 00:12:56 GMT</pubDate>
   </item>
  <item>
    <title>defensio</title>
    <link>http://pypi.python.org/pypi/defensio/0.9.1</link>
    <description>new release</description>
    <pubDate>20 Jan 2010 00:12:56 GMT</pubDate>
   </item>
  <item>
    <title>iconv</title>
    <link>http://pypi.python.org/pypi/iconv/1.0</link>
    <description>update platform, classifiers</description>
    <pubDate>20 Jan 2010 00:16:22 GMT</pubDate>
   </item>
  </channel>
</rss>

David



From mal at egenix.com  Wed Jan 20 10:45:36 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 20 Jan 2010 10:45:36 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B553366.2090809@holdenweb.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>
	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>
	<4B553366.2090809@holdenweb.com>
Message-ID: <4B56D0C0.2080703@egenix.com>

Steve Holden wrote:
> None that I am aware of, but Martin is the one who's been making changes
> most recently. I don't think there's been any input from Van on this
> yet, but I've been busy and may have forgotten or missed something.

Thanks.

As far as I can tell, the text on the PyPI registration page hasn't
changed since we last discussed the problem.

Since there is currently discussion going on about setting up
PyPI mirrors that are not maintained by the PSF, I think that
we need to do something about the licensing terms soonish, esp.
since most package authors who have uploaded things to PyPI
will not be aware of any changes to the terms and conditions of
using PyPI.

Any such change will have to be made more visible on the site
and the Python announcement list than is currently done (you only
see the text if you click on "register" on PyPI - which, of course,
already registered users will unlikely do on a regular basis :-).

Regards,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 20 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


> regards
>  Steve
> 
> M.-A. Lemburg wrote:
>> Hi Steve,
>>
>> has there been any progress on this ?
>>
>> M.-A. Lemburg wrote:
>>> Steve Holden, Chairman, PSF wrote:
>>>> Adding a Google-like clause might make us seem less Draconian.
>>> Here's a proposal for a less controversial text based on the Google
>>> terms:
>>>
>>> """
>>> PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to
>>> PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following:
>>>
>>> 1. Content is restricted to Python packages and related information only.
>>>
>>> 2. Any content uploaded to PyPI is provided on a non-confidential basis.
>>>
>>> 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce,
>>> distribute, transmit, display, perform, and publish the content, including in digital form. This
>>> licence is for the sole purpose of enabling the PSF to display, distribute and promote the content
>>> on PyPI.
>>>
>>> 4. I represent and warrant that I have complied with all government regulations concerning the
>>> transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if
>>> I am subject to United States law, I represent and warrant that I have obtained the proper
>>> governmental authorization for the export of the content I upload. I further affirm that any content
>>> I provide is not intended for use by a government end-user as defined in part 772 of the United
>>> States Export Administration Regulations.
>>> """
>>>
>>> The general terms on the python.org legal page would have to be
>>> changed in the same way.
>>
>> I've attached a message explaining some of the reasons for part 4.
>>
>> Thanks,
>>
>>
>> ------------------------------------------------------------------------
>>
>> Subject:
>> Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
>> From:
>> "M.-A. Lemburg" <mal at egenix.com>
>> Date:
>> Fri, 11 Dec 2009 01:45:10 +0100
>> To:
>> Terry Reedy <tjreedy at udel.edu>
>>
>> To:
>> Terry Reedy <tjreedy at udel.edu>
>> CC:
>> catalog-sig at python.org, VanL <van at python.org>
>>
>>
>> Terry Reedy wrote:
>>> M.-A. Lemburg wrote:
>>>> Steve Holden, Chairman, PSF wrote:
>>>>> Adding a Google-like clause might make us seem less Draconian.
>>>> Here's a proposal for a less controversial text based on the Google
>>>> terms:
>>> I like the third part better.
>>
>> Thanks.
>>
>>>> """
>>>> PyPI is a service provided by the PSF. In order to be able to
>>>> distribute the content you upload to
>>>> PyPI to web site users, the PSF asks you to agree to and affirmatively
>>>> acknowledge the following:
>>>>
>>>> 1. Content is restricted to Python packages and related information only.
>>>>
>>>> 2. Any content uploaded to PyPI is provided on a non-confidential basis.
>>>>
>>>> 3. The PSF is granted an irrevocable, worldwide, royalty-free,
>>>> nonexclusive license to reproduce,
>>>> distribute, transmit, display, perform, and publish the content,
>>>> including in digital form. This
>>>> licence is for the sole purpose of enabling the PSF to display,
>>>> distribute and promote the content
>>>> on PyPI.
>>>>
>>>> 4. I represent and warrant that I have complied with all government
>>>> regulations concerning the
>>>> transfer or export of any content I upload to the PyPI servers in The
>>>> Netherlands. In particular, if
>>>> I am subject to United States law, I represent and warrant that I have
>>>> obtained the proper
>>>> governmental authorization for the export of the content I upload. I
>>>> further affirm that any content
>>>> I provide is not intended for use by a government end-user as defined
>>>> in part 772 of the United
>>>> States Export Administration Regulations.
>>>> """
>>> The fourth section might scare people off without further explanation
>>> somewhere, as it could be taken to imply that people have to get a US
>>> gov permit to upload, which almost no one has done. If this is only
>>> about crypto software, it should say so. I do not understand the last
>>> sentence at all as open-source licenses do not usually exclude specific
>>> users. I cannot affirm something that is complete gobble talk to me.
>>
>> The clause has three parts:
>>
>>  a) "I represent and warrant that I have complied with all government regulations concerning the
>> transfer or export of any content I upload to the PyPI servers in The Netherlands."
>>
>> This part is written in a general way and is needed to
>> cover export regulations which may be imposed by the country
>> of the uploader when uploading (exporting) applications to
>> a server in the The Netherlands.
>>
>> For many countries these export regulations are variants
>> of the things laid out in the Wassenaar Arrangement which
>> covers crypto code, but also other software technologies
>> that may be considered dual-use:
>>
>> http://www.wassenaar.org/
>> in particular:
>> http://www.wassenaar.org/controllists/2009/WA-LIST%20%2809%29%201/WA-LIST%20%2809%29%201.pdf
>>
>> Most software will fall under the "GENERAL SOFTWARE NOTE"
>> (with some special rules for crypto software), but countries
>> may still implement additional rules such as the ones currently
>> imposed by the US (you have to send them an email with the link
>> to the download location - see
>> http://www.bis.doc.gov/encryption/pubavailencsourcecodenofify.html).
>>
>> Since the exact regulations depend on the country from where
>> the code is uploaded, the clause can't be more specific.
>>
>> I added the location of the servers to the original clause to
>> make the export nature of the upload more specific.
>>
>>  b) "In particular, if I am subject to United States law, I represent and warrant that I have
>> obtained the proper governmental authorization for the export of the content I upload."
>>
>> This part only applies to US uploaders.
>>
>> Note that the US regulations have a subtle detail: they apply to
>> all US-origin content. E.g. if you export some dual-use system software
>> written in the US from Germany to Cuba, the US can put you on their
>> embargo list.
>>
>>  c)  "I further affirm that any content I provide is not intended for use by a government end-user
>> as defined in part 772 of the United States Export Administration Regulations."
>>
>> This part applies to all uploaders. The restriction appears to be
>> a super-set of the embargo restrictions for various individuals -
>> most of those are government end-users.
>>
>> I find that clause too board as well, since it prevents government
>> users in general to use PyPI packages.
>>
>> Furthermore, the embargo lists also includes companies and, of course,
>> whole countries, which this clause does not cover. See e.g.
>> EU: http://ec.europa.eu/external_relations/cfsp/sanctions/docs/measures_en.pdf
>> US: http://www.bis.doc.gov/news/2009/2009-fpr.pdf
>> (note how e.g. Cuba is on the US list, but not on the EU list)
>>
>> I'm not sure why the clause is needed. Perhaps Van could clarify
>> this.
>>
>> IMHO, part a) already covers everything that is needed w/r to
>> export restrictions.
>>
>> All this with the usual IANAL disclaimer. I've read a lot on these
>> things when we started shipping a pyOpenSSL distribution. Some of the
>> things I found are listed above.
>>
> 
> 


From steve at holdenweb.com  Wed Jan 20 16:26:00 2010
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 20 Jan 2010 10:26:00 -0500
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B56D0C0.2080703@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>
	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>
	<4B553366.2090809@holdenweb.com> <4B56D0C0.2080703@egenix.com>
Message-ID: <4B572088.7020501@holdenweb.com>

Agreed. Until this issue is resolved we can't allow (public) third-party
mirrors. Given the recent adverse reactions to PyPi changes we should be
careful not to cause any further offense.

regards
 Steve

M.-A. Lemburg wrote:
> Steve Holden wrote:
>> None that I am aware of, but Martin is the one who's been making changes
>> most recently. I don't think there's been any input from Van on this
>> yet, but I've been busy and may have forgotten or missed something.
> 
> Thanks.
> 
> As far as I can tell, the text on the PyPI registration page hasn't
> changed since we last discussed the problem.
> 
> Since there is currently discussion going on about setting up
> PyPI mirrors that are not maintained by the PSF, I think that
> we need to do something about the licensing terms soonish, esp.
> since most package authors who have uploaded things to PyPI
> will not be aware of any changes to the terms and conditions of
> using PyPI.
> 
> Any such change will have to be made more visible on the site
> and the Python announcement list than is currently done (you only
> see the text if you click on "register" on PyPI - which, of course,
> already registered users will unlikely do on a regular basis :-).
> 
> Regards,


-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010  http://us.pycon.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/

From mal at egenix.com  Wed Jan 20 20:06:19 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 20 Jan 2010 20:06:19 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B572088.7020501@holdenweb.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com> <4B572088.7020501@holdenweb.com>
Message-ID: <4B57542B.5060305@egenix.com>

Steve Holden wrote:
> Agreed. Until this issue is resolved we can't allow (public) third-party
> mirrors. Given the recent adverse reactions to PyPi changes we should be
> careful not to cause any further offense.

Perhaps the PSF should start creating a list of official public
mirrors. We would then need to add a mention of this to the PyPI
usage license I posted.

Setting up an official mirror would then require approval of the PSF
and allow the PSF to monitor and maintain a good quality service.
Package manager tools could then use the list of official mirrors
to find a nearby mirror for downloading the package information.

Regards,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 20 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


> regards
>  Steve
> 
> M.-A. Lemburg wrote:
>> Steve Holden wrote:
>>> None that I am aware of, but Martin is the one who's been making changes
>>> most recently. I don't think there's been any input from Van on this
>>> yet, but I've been busy and may have forgotten or missed something.
>>
>> Thanks.
>>
>> As far as I can tell, the text on the PyPI registration page hasn't
>> changed since we last discussed the problem.
>>
>> Since there is currently discussion going on about setting up
>> PyPI mirrors that are not maintained by the PSF, I think that
>> we need to do something about the licensing terms soonish, esp.
>> since most package authors who have uploaded things to PyPI
>> will not be aware of any changes to the terms and conditions of
>> using PyPI.
>>
>> Any such change will have to be made more visible on the site
>> and the Python announcement list than is currently done (you only
>> see the text if you click on "register" on PyPI - which, of course,
>> already registered users will unlikely do on a regular basis :-).
>>
>> Regards,
> 
> 


From ziade.tarek at gmail.com  Wed Jan 20 22:34:08 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 20 Jan 2010 22:34:08 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B57542B.5060305@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B1D61C3.7030705@python.org>
	<4B1D7601.5040608@egenix.com> <4B1DB9BE.5040000@python.org>
	<4B1FE7B1.9030608@egenix.com> <4B54D896.2060108@egenix.com>
	<4B553366.2090809@holdenweb.com> <4B56D0C0.2080703@egenix.com>
	<4B572088.7020501@holdenweb.com> <4B57542B.5060305@egenix.com>
Message-ID: <94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>

On Wed, Jan 20, 2010 at 8:06 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> Steve Holden wrote:
>> Agreed. Until this issue is resolved we can't allow (public) third-party
>> mirrors. Given the recent adverse reactions to PyPi changes we should be
>> careful not to cause any further offense.
>
> Perhaps the PSF should start creating a list of official public
> mirrors. We would then need to add a mention of this to the PyPI
> usage license I posted.
>
> Setting up an official mirror would then require approval of the PSF
> and allow the PSF to monitor and maintain a good quality service.
> Package manager tools could then use the list of official mirrors
> to find a nearby mirror for downloading the package information.

That's basically what I've proposed in PEP 381 :

1/ the mirror maintainer proposes its mirror in the Catalog mailing-list
2/ if it's accepted, it's added in the list of IPs in the pypi DNS
3/ client tools like PIP develop a geloc feature to use the nearest
mirror, by getting the list of IPs

http://www.python.org/dev/peps/pep-0381/#how-a-client-can-use-pypi-and-its-mirrors

I could add in the PEP the fact that the mirror has to be accepted by the PSF

Tarek

-- 
Tarek Ziad? | http://ziade.org

From ziade.tarek at gmail.com  Wed Jan 20 22:35:59 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 20 Jan 2010 22:35:59 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B1D7601.5040608@egenix.com>
	<4B1DB9BE.5040000@python.org> <4B1FE7B1.9030608@egenix.com>
	<4B54D896.2060108@egenix.com> <4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com> <4B572088.7020501@holdenweb.com>
	<4B57542B.5060305@egenix.com>
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
Message-ID: <94bdd2611001201335i743a29afq673be320d2886202@mail.gmail.com>

On Wed, Jan 20, 2010 at 10:34 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
[..]
> http://www.python.org/dev/peps/pep-0381/#how-a-client-can-use-pypi-and-its-mirrors
>
> I could add in the PEP the fact that the mirror has to be accepted by the PSF

See also : http://www.python.org/dev/peps/pep-0381/#mirror-listing-and-registering

-- 
Tarek Ziad? | http://ziade.org

From ziade.tarek at gmail.com  Wed Jan 20 22:40:06 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 20 Jan 2010 22:40:06 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <94bdd2611001201335i743a29afq673be320d2886202@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B1DB9BE.5040000@python.org>
	<4B1FE7B1.9030608@egenix.com> <4B54D896.2060108@egenix.com>
	<4B553366.2090809@holdenweb.com> <4B56D0C0.2080703@egenix.com>
	<4B572088.7020501@holdenweb.com> <4B57542B.5060305@egenix.com>
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
	<94bdd2611001201335i743a29afq673be320d2886202@mail.gmail.com>
Message-ID: <94bdd2611001201340v4c8e7accp6c709bcafacc8947@mail.gmail.com>

On Wed, Jan 20, 2010 at 10:35 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Wed, Jan 20, 2010 at 10:34 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> [..]
>> http://www.python.org/dev/peps/pep-0381/#how-a-client-can-use-pypi-and-its-mirrors
>>
>> I could add in the PEP the fact that the mirror has to be accepted by the PSF
>
> See also : http://www.python.org/dev/peps/pep-0381/#mirror-listing-and-registering

Notice also that the list of mirrors is already implemented in PyPI :
http://pypi.python.org/mirrors

Sorry for the 3 mails-in-the-row :)

From martin at v.loewis.de  Wed Jan 20 22:42:48 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 20 Jan 2010 22:42:48 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B572088.7020501@holdenweb.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com> <4B572088.7020501@holdenweb.com>
Message-ID: <4B5778D8.9090402@v.loewis.de>

Steve Holden wrote:
> Agreed. Until this issue is resolved we can't allow (public) third-party
> mirrors. Given the recent adverse reactions to PyPi changes we should be
> careful not to cause any further offense.

I quite disagree on that statement; I see the issue of mirrors as
completely unrelated, and I also don't think the current issue is
causing any offense.

Regards,
Martin

From mal at egenix.com  Wed Jan 20 22:46:49 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 20 Jan 2010 22:46:49 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <94bdd2611001201335i743a29afq673be320d2886202@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>
	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>
	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>
	<4B572088.7020501@holdenweb.com>	<4B57542B.5060305@egenix.com>	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
	<94bdd2611001201335i743a29afq673be320d2886202@mail.gmail.com>
Message-ID: <4B5779C9.3070600@egenix.com>

Tarek Ziad? wrote:
> On Wed, Jan 20, 2010 at 8:06 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> > Steve Holden wrote:
>>> >> Agreed. Until this issue is resolved we can't allow (public) third-party
>>> >> mirrors. Given the recent adverse reactions to PyPi changes we should be
>>> >> careful not to cause any further offense.
>> >
>> > Perhaps the PSF should start creating a list of official public
>> > mirrors. We would then need to add a mention of this to the PyPI
>> > usage license I posted.
>> >
>> > Setting up an official mirror would then require approval of the PSF
>> > and allow the PSF to monitor and maintain a good quality service.
>> > Package manager tools could then use the list of official mirrors
>> > to find a nearby mirror for downloading the package information.
> That's basically what I've proposed in PEP 381 :
> 
> 1/ the mirror maintainer proposes its mirror in the Catalog mailing-list
> 2/ if it's accepted, it's added in the list of IPs in the pypi DNS
> 3/ client tools like PIP develop a geloc feature to use the nearest
> mirror, by getting the list of IPs
> 
> http://www.python.org/dev/peps/pep-0381/#how-a-client-can-use-pypi-and-its-mirrors
> 
> I could add in the PEP the fact that the mirror has to be accepted by the PSF

Tarek Ziad? wrote:
> On Wed, Jan 20, 2010 at 10:34 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> [..]
>> http://www.python.org/dev/peps/pep-0381/#how-a-client-can-use-pypi-and-its-mirrors
>>
>> I could add in the PEP the fact that the mirror has to be accepted by the PSF
> 
> See also : http://www.python.org/dev/peps/pep-0381/#mirror-listing-and-registering

Any such addition will have to go by the PSF since the PSF runs the
official PyPI index.

Note that this is only required for PyPI mirrors that are made public,
ie. redistribute the PyPI data without the PSF being directly involved.

Background: the change recently introduced on the PyPI registration
page does cover such redistributions by any user of PyPI, but that
change is controversial and, what's more important, does not cover
data uploaded to PyPI by users who registered before the change.

I'd like to get the wording on the terms changed to make the PSF
the only partner in the agreement and let it control the redistribution
terms - much like we do in the Python contribution forms.

Just to clarify: a private mirror would basically just be standard use
of the PyPI data without the redistribution part, so that use is fine
in all cases.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 20 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Wed Jan 20 22:54:43 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 20 Jan 2010 22:54:43 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5778D8.9090402@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>
	<4B572088.7020501@holdenweb.com> <4B5778D8.9090402@v.loewis.de>
Message-ID: <4B577BA3.7070606@egenix.com>

"Martin v. L?wis" wrote:
> Steve Holden wrote:
>> Agreed. Until this issue is resolved we can't allow (public) third-party
>> mirrors. Given the recent adverse reactions to PyPi changes we should be
>> careful not to cause any further offense.
> 
> I quite disagree on that statement; I see the issue of mirrors as
> completely unrelated, and I also don't think the current issue is
> causing any offense.

Any public mirror would trigger a redistribution of PyPI uploaded
data without the PSF or the uploading PyPI user owning the material
having a say about the redistribution terms.

This may sound harmless at first, but if we get more VyperLogix
folks setting up PyPI mirrors, I'm sure that a lot of developers
will not be happy about the PSF being careless about who is allowed
to redistribute the PyPI data.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 20 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From martin at v.loewis.de  Wed Jan 20 22:56:25 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 20 Jan 2010 22:56:25 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI
	usage	agreement
In-Reply-To: <94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>
	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>
	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>
	<4B57542B.5060305@egenix.com>
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
Message-ID: <4B577C09.3050201@v.loewis.de>

> I could add in the PEP the fact that the mirror has to be accepted by the PSF

Please don't: I think first the PSF would have to claim any interest in
micro-managing PyPI. I don't think it's their mission to do so.

Regards,
Martin

From martin at v.loewis.de  Wed Jan 20 22:58:38 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 20 Jan 2010 22:58:38 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B5779C9.3070600@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B57542B.5060305@egenix.com>	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>	<94bdd2611001201335i743a29afq673be320d2886202@mail.gmail.com>
	<4B5779C9.3070600@egenix.com>
Message-ID: <4B577C8E.3000605@v.loewis.de>

> Any such addition will have to go by the PSF since the PSF runs the
> official PyPI index.

I don't think this is actually true. That the PSF runs the services
does *not* mean that changes have to be approved by the PSF. In fact,
many volunteers constantly change www.python.org in many ways, without
any per-change approval.

Regards,
Martin

From martin at v.loewis.de  Wed Jan 20 23:00:09 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 20 Jan 2010 23:00:09 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B577BA3.7070606@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>
	<4B5778D8.9090402@v.loewis.de> <4B577BA3.7070606@egenix.com>
Message-ID: <4B577CE9.2050809@v.loewis.de>

> This may sound harmless at first, but if we get more VyperLogix
> folks setting up PyPI mirrors, I'm sure that a lot of developers
> will not be happy about the PSF being careless about who is allowed
> to redistribute the PyPI data.

See Tarek's response: not running mirror changes to the PSF board
doesn't mean that addition of mirrors would be done carelessly.
In fact, the whole point of PEP 381 is to do careful mirroring.

Regards,
Martin


From steve at holdenweb.com  Wed Jan 20 23:01:18 2010
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 20 Jan 2010 17:01:18 -0500
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B1D61C3.7030705@python.org>	
	<4B1D7601.5040608@egenix.com> <4B1DB9BE.5040000@python.org>	
	<4B1FE7B1.9030608@egenix.com> <4B54D896.2060108@egenix.com>	
	<4B553366.2090809@holdenweb.com> <4B56D0C0.2080703@egenix.com>	
	<4B572088.7020501@holdenweb.com> <4B57542B.5060305@egenix.com>
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
Message-ID: <4B577D2E.6060606@holdenweb.com>

Tarek Ziad? wrote:
> On Wed, Jan 20, 2010 at 8:06 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> Steve Holden wrote:
>>> Agreed. Until this issue is resolved we can't allow (public) third-party
>>> mirrors. Given the recent adverse reactions to PyPi changes we should be
>>> careful not to cause any further offense.
>> Perhaps the PSF should start creating a list of official public
>> mirrors. We would then need to add a mention of this to the PyPI
>> usage license I posted.
>>
>> Setting up an official mirror would then require approval of the PSF
>> and allow the PSF to monitor and maintain a good quality service.
>> Package manager tools could then use the list of official mirrors
>> to find a nearby mirror for downloading the package information.
> 
> That's basically what I've proposed in PEP 381 :
> 
> 1/ the mirror maintainer proposes its mirror in the Catalog mailing-list
> 2/ if it's accepted, it's added in the list of IPs in the pypi DNS
> 3/ client tools like PIP develop a geloc feature to use the nearest
> mirror, by getting the list of IPs
> 
> http://www.python.org/dev/peps/pep-0381/#how-a-client-can-use-pypi-and-its-mirrors
> 
> I could add in the PEP the fact that the mirror has to be accepted by the PSF
> 
> Tarek
> 
Given the situation last time we had mirrors (though not of PyPi) it
would be useful to have some automated monitoring. They get stale very
quickly if nobody stays on top of them, and a stale mirror is worse than
no mirror at all.

What can we do to establish and maintain the right attitude?

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010  http://us.pycon.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/

From ziade.tarek at gmail.com  Wed Jan 20 23:13:51 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 20 Jan 2010 23:13:51 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B577D2E.6060606@holdenweb.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B1DB9BE.5040000@python.org>
	<4B1FE7B1.9030608@egenix.com> <4B54D896.2060108@egenix.com>
	<4B553366.2090809@holdenweb.com> <4B56D0C0.2080703@egenix.com>
	<4B572088.7020501@holdenweb.com> <4B57542B.5060305@egenix.com>
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
	<4B577D2E.6060606@holdenweb.com>
Message-ID: <94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>

On Wed, Jan 20, 2010 at 11:01 PM, Steve Holden <steve at holdenweb.com> wrote:
[..]
>>
> Given the situation last time we had mirrors (though not of PyPi) it
> would be useful to have some automated monitoring. They get stale very
> quickly if nobody stays on top of them, and a stale mirror is worse than
> no mirror at all.
>
> What can we do to establish and maintain the right attitude?

Since each mirror has to maintain a "last modified" page that is the
date of the last
update, we could discard mirrors that are lagging too much and too often.

Of course, there's also a human dimension : we suppose that the people
running the mirror are people we can trust because they can
technically do malicious things in the mirror since we don't really
have any real protection (*yet*). I don't know how this trust can be
measured...

Regards
Tarek

From mal at egenix.com  Wed Jan 20 23:15:27 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 20 Jan 2010 23:15:27 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B577CE9.2050809@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>
	<4B577BA3.7070606@egenix.com> <4B577CE9.2050809@v.loewis.de>
Message-ID: <4B57807F.2050303@egenix.com>

"Martin v. L?wis" wrote:
>> This may sound harmless at first, but if we get more VyperLogix
>> folks setting up PyPI mirrors, I'm sure that a lot of developers
>> will not be happy about the PSF being careless about who is allowed
>> to redistribute the PyPI data.
> 
> See Tarek's response: not running mirror changes to the PSF board
> doesn't mean that addition of mirrors would be done carelessly.
> In fact, the whole point of PEP 381 is to do careful mirroring.

Sure, the PEP can be used as basis for the decision process, but
someone still has to make the decision to add a mirror or not
and these people should be appointed to by the PSF - much like we
have an infrastructure committee to see after the python.org site.

The situation is a lot like with the Python trademarks:
The good guys always come and ask for permission. The bad guys
don't. In order to go after them we need a clear set of
rules for setting up and running a PyPI mirror and disallowing
setups that don't follow these rules.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 20 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ben+python at benfinney.id.au  Wed Jan 20 23:15:14 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 21 Jan 2010 09:15:14 +1100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>
	<4B193350.7090609@python.org> <4B1961D1.7010405@egenix.com>
	<019d01ca7518$2dba00f0$892e02d0$@net> <4B1D5CCB.4000309@egenix.com>
	<4B1D61C3.7030705@python.org> <4B1D7601.5040608@egenix.com>
	<4B1DB9BE.5040000@python.org> <4B1FE7B1.9030608@egenix.com>
	<4B54D896.2060108@egenix.com> <4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com> <4B572088.7020501@holdenweb.com>
	<4B5778D8.9090402@v.loewis.de>
Message-ID: <87y6jse83x.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> Steve Holden wrote:
> > Agreed. Until this issue is resolved we can't allow (public) third-party
> > mirrors. Given the recent adverse reactions to PyPi changes we should be
> > careful not to cause any further offense.
>
> I quite disagree on that statement; I see the issue of mirrors as
> completely unrelated, and I also don't think the current issue is
> causing any offense.

I also don't think the current issue is causing offense. What I *do*
think is that the current terms are over-reaching and need to be
changed, as outlined near the beginning of this thread.

-- 
 \        ?Always code as if the guy who ends up maintaining your code |
  `\     will be a violent psychopath who knows where you live.? ?John |
_o__)                                                         F. Woods |
Ben Finney


From martin at v.loewis.de  Wed Jan 20 23:38:12 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 20 Jan 2010 23:38:12 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B577D2E.6060606@holdenweb.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B1D61C3.7030705@python.org>		<4B1D7601.5040608@egenix.com>
	<4B1DB9BE.5040000@python.org>		<4B1FE7B1.9030608@egenix.com>
	<4B54D896.2060108@egenix.com>		<4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com>		<4B572088.7020501@holdenweb.com>
	<4B57542B.5060305@egenix.com>	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
	<4B577D2E.6060606@holdenweb.com>
Message-ID: <4B5785D4.7020704@v.loewis.de>

> Given the situation last time we had mirrors (though not of PyPi) it
> would be useful to have some automated monitoring. They get stale very
> quickly if nobody stays on top of them, and a stale mirror is worse than
> no mirror at all.
> 
> What can we do to establish and maintain the right attitude?

PEP 381 provides the right attitude.

Regards,
Martin

From martin at v.loewis.de  Wed Jan 20 23:40:48 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 20 Jan 2010 23:40:48 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI
	usage	agreement
In-Reply-To: <94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>
	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>
	<4B57542B.5060305@egenix.com>	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>	<4B577D2E.6060606@holdenweb.com>
	<94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>
Message-ID: <4B578670.6010203@v.loewis.de>

> Of course, there's also a human dimension : we suppose that the people
> running the mirror are people we can trust because they can
> technically do malicious things in the mirror since we don't really
> have any real protection (*yet*).

That's not true: users of mirrors can verify that the mirrors are
authentic. Neither can malicious operators modify the contents of
their mirrors without clients noticing, nor can careless mirror
operators threaten the integrity of a mirror even assuming somebody
breaks into the mirror.

Regards,
Martin

From martin at v.loewis.de  Wed Jan 20 23:42:53 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 20 Jan 2010 23:42:53 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B57807F.2050303@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>
	<4B577BA3.7070606@egenix.com> <4B577CE9.2050809@v.loewis.de>
	<4B57807F.2050303@egenix.com>
Message-ID: <4B5786ED.2020708@v.loewis.de>

> Sure, the PEP can be used as basis for the decision process, but
> someone still has to make the decision to add a mirror or not
> and these people should be appointed to by the PSF - much like we
> have an infrastructure committee to see after the python.org site.
> 
> The situation is a lot like with the Python trademarks:
> The good guys always come and ask for permission. The bad guys
> don't. 

With PEP 381, the bad guys won't get any attention. You can already
run as many public mirrors of PyPI as you want, but nobody will notice.

To become an official mirror, you have to ask for permission already;
see the PEP.

> In order to go after them we need a clear set of
> rules for setting up and running a PyPI mirror and disallowing
> setups that don't follow these rules.

See the PEP.

Regards,
Martin


From ziade.tarek at gmail.com  Thu Jan 21 00:05:40 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 21 Jan 2010 00:05:40 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B578670.6010203@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B54D896.2060108@egenix.com>
	<4B553366.2090809@holdenweb.com> <4B56D0C0.2080703@egenix.com>
	<4B572088.7020501@holdenweb.com> <4B57542B.5060305@egenix.com>
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
	<4B577D2E.6060606@holdenweb.com>
	<94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>
	<4B578670.6010203@v.loewis.de>
Message-ID: <94bdd2611001201505p839dc7arb4ec0aeae7b84194@mail.gmail.com>

2010/1/20 "Martin v. L?wis" <martin at v.loewis.de>:
>> Of course, there's also a human dimension : we suppose that the people
>> running the mirror are people we can trust because they can
>> technically do malicious things in the mirror since we don't really
>> have any real protection (*yet*).
>
> That's not true: users of mirrors can verify that the mirrors are
> authentic. Neither can malicious operators modify the contents of
> their mirrors without clients noticing, nor can careless mirror
> operators threaten the integrity of a mirror even assuming somebody
> breaks into the mirror.

But users can't verify that the archive they download using tools like
easy_install are the real ones.

If I am a bad guy and I run a mirror, I can change a setup.py file in
an archive and
make it do malicious things on the computer, and let easy_install
execute it for me.
The only verification done is the md5 hash on the file, which can be
changed on the mirror (nothing prevents the mirror to compute its own
MD5 fragments in the download URLs)

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Thu Jan 21 00:35:27 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 21 Jan 2010 00:35:27 +0100
Subject: [Catalog-sig] PEP 381: server signatures (Was: Troubled by
 changes to PyPI usage agreement)
In-Reply-To: <94bdd2611001201505p839dc7arb4ec0aeae7b84194@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B54D896.2060108@egenix.com>	
	<4B553366.2090809@holdenweb.com> <4B56D0C0.2080703@egenix.com>	
	<4B572088.7020501@holdenweb.com> <4B57542B.5060305@egenix.com>	
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>	
	<4B577D2E.6060606@holdenweb.com>	
	<94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>	
	<4B578670.6010203@v.loewis.de>
	<94bdd2611001201505p839dc7arb4ec0aeae7b84194@mail.gmail.com>
Message-ID: <4B57933F.9000503@v.loewis.de>

> The only verification done is the md5 hash on the file, which can be
> changed on the mirror (nothing prevents the mirror to compute its own
> MD5 fragments in the download URLs)

That's not true. Changing the MD-5 would require to change the simple
page, and that in turn would break the server signature to that page.

In case you are unaware of the server signature, please have a look at

http://mail.python.org/pipermail/catalog-sig/2009-March/002018.html

I'd appreciate if that would be added to the PEP.

Regards,
Martin

From ziade.tarek at gmail.com  Thu Jan 21 00:41:13 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 21 Jan 2010 00:41:13 +0100
Subject: [Catalog-sig] PEP 381: server signatures (Was: Troubled by
	changes to PyPI usage agreement)
In-Reply-To: <4B57933F.9000503@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B56D0C0.2080703@egenix.com>
	<4B572088.7020501@holdenweb.com> <4B57542B.5060305@egenix.com>
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
	<4B577D2E.6060606@holdenweb.com>
	<94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>
	<4B578670.6010203@v.loewis.de>
	<94bdd2611001201505p839dc7arb4ec0aeae7b84194@mail.gmail.com>
	<4B57933F.9000503@v.loewis.de>
Message-ID: <94bdd2611001201541v27ebd281hda5ba57bbaa4c4d8@mail.gmail.com>

2010/1/21 "Martin v. L?wis" <martin at v.loewis.de>:
>> The only verification done is the md5 hash on the file, which can be
>> changed on the mirror (nothing prevents the mirror to compute its own
>> MD5 fragments in the download URLs)
>
> That's not true. Changing the MD-5 would require to change the simple
> page, and that in turn would break the server signature to that page.
>
> In case you are unaware of the server signature, please have a look at
>
> http://mail.python.org/pipermail/catalog-sig/2009-March/002018.html
>

I forgot about that one, thanks for the memories

> I'd appreciate if that would be added to the PEP.
>

Yes definitely, I'll do that

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From steve at holdenweb.com  Thu Jan 21 03:30:58 2010
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 20 Jan 2010 21:30:58 -0500
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <94bdd2611001201505p839dc7arb4ec0aeae7b84194@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B54D896.2060108@egenix.com>	
	<4B553366.2090809@holdenweb.com> <4B56D0C0.2080703@egenix.com>	
	<4B572088.7020501@holdenweb.com> <4B57542B.5060305@egenix.com>	
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>	
	<4B577D2E.6060606@holdenweb.com>	
	<94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>	
	<4B578670.6010203@v.loewis.de>
	<94bdd2611001201505p839dc7arb4ec0aeae7b84194@mail.gmail.com>
Message-ID: <4B57BC62.4090401@holdenweb.com>

Tarek Ziad? wrote:
> 2010/1/20 "Martin v. L?wis" <martin at v.loewis.de>:
>>> Of course, there's also a human dimension : we suppose that the people
>>> running the mirror are people we can trust because they can
>>> technically do malicious things in the mirror since we don't really
>>> have any real protection (*yet*).
>> That's not true: users of mirrors can verify that the mirrors are
>> authentic. Neither can malicious operators modify the contents of
>> their mirrors without clients noticing, nor can careless mirror
>> operators threaten the integrity of a mirror even assuming somebody
>> breaks into the mirror.
> 
> But users can't verify that the archive they download using tools like
> easy_install are the real ones.
> 
> If I am a bad guy and I run a mirror, I can change a setup.py file in
> an archive and
> make it do malicious things on the computer, and let easy_install
> execute it for me.
> The only verification done is the md5 hash on the file, which can be
> changed on the mirror (nothing prevents the mirror to compute its own
> MD5 fragments in the download URLs)
> 
> Regards
> Tarek
> 
I have in the past suggested that we consider hosting services at
diverse places. I'd have thought this was a prima facie case for
distributed hosting facilities. If we have that, we have no need for
mirrors, but instead for systems management. I know of at least three
reputable US hosting companies who I am pretty sure would help, and a
major academic hosting organization too. Also maybe Snakebite might be
another location?

This is an infrastructure committee task really. Brett? Martin?

Brett: maybe your closing report to the board could summarize the
overall hosting situation?

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010  http://us.pycon.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/

From mal at egenix.com  Thu Jan 21 11:59:36 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2010 11:59:36 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5786ED.2020708@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>
	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de>
Message-ID: <4B583398.9090509@egenix.com>

"Martin v. L?wis" wrote:
>> Sure, the PEP can be used as basis for the decision process, but
>> someone still has to make the decision to add a mirror or not
>> and these people should be appointed to by the PSF - much like we
>> have an infrastructure committee to see after the python.org site.
>>
>> The situation is a lot like with the Python trademarks:
>> The good guys always come and ask for permission. The bad guys
>> don't. 
> 
> With PEP 381, the bad guys won't get any attention. You can already
> run as many public mirrors of PyPI as you want, but nobody will notice.
> 
> To become an official mirror, you have to ask for permission already;
> see the PEP.
> 
>> In order to go after them we need a clear set of
>> rules for setting up and running a PyPI mirror and disallowing
>> setups that don't follow these rules.
> 
> See the PEP.

Like I said: the PEP can be used to document the technical requirements
of being accepted as official mirror, but it doesn't cover any
of the legal requirements the PSF will need to put in place in
order to prevent unofficial mirrors which use the content to
e.g. increase their page rank, add advertisement and other
drive-by revenue, misrepresent authorship, add malware to popular
packages, etc.

Such requirements are not within the scope of a PEP. The decision
process itself is also not within the scope of the PEP. It only
outlines the technical details of official mirrors.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 21 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Thu Jan 21 12:24:33 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2010 12:24:33 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI
	usage	agreement
In-Reply-To: <4B57BC62.4090401@holdenweb.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B54D896.2060108@egenix.com>		<4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com>		<4B572088.7020501@holdenweb.com>
	<4B57542B.5060305@egenix.com>		<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>		<4B577D2E.6060606@holdenweb.com>		<94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>		<4B578670.6010203@v.loewis.de>	<94bdd2611001201505p839dc7arb4ec0aeae7b84194@mail.gmail.com>
	<4B57BC62.4090401@holdenweb.com>
Message-ID: <4B583971.4060704@egenix.com>

Steve Holden wrote:
> Tarek Ziad? wrote:
>> 2010/1/20 "Martin v. L?wis" <martin at v.loewis.de>:
>>>> Of course, there's also a human dimension : we suppose that the people
>>>> running the mirror are people we can trust because they can
>>>> technically do malicious things in the mirror since we don't really
>>>> have any real protection (*yet*).
>>> That's not true: users of mirrors can verify that the mirrors are
>>> authentic. Neither can malicious operators modify the contents of
>>> their mirrors without clients noticing, nor can careless mirror
>>> operators threaten the integrity of a mirror even assuming somebody
>>> breaks into the mirror.
>>
>> But users can't verify that the archive they download using tools like
>> easy_install are the real ones.
>>
>> If I am a bad guy and I run a mirror, I can change a setup.py file in
>> an archive and
>> make it do malicious things on the computer, and let easy_install
>> execute it for me.
>> The only verification done is the md5 hash on the file, which can be
>> changed on the mirror (nothing prevents the mirror to compute its own
>> MD5 fragments in the download URLs)
>>
>> Regards
>> Tarek
>>
> I have in the past suggested that we consider hosting services at
> diverse places. I'd have thought this was a prima facie case for
> distributed hosting facilities. If we have that, we have no need for
> mirrors, but instead for systems management. I know of at least three
> reputable US hosting companies who I am pretty sure would help, and a
> major academic hosting organization too. Also maybe Snakebite might be
> another location?

That would be my preference as well.

Services like Amazon CloudFront or Akamai could easily scale up to
millions of users.

Since the PyPI data is already available as set of static files
(at least from looking at the simple/ index), pushing this through
such content delivery systems should easily be possible.

Such a setup would avoid all the legalese issues, since the PSF
would run the infrastructure and also make monitoring the service
a lot easier.

Moreover, all the complicated stuff like on-demand content
distribution is handled by those services automatically.

> This is an infrastructure committee task really. Brett? Martin?
> 
> Brett: maybe your closing report to the board could summarize the
> overall hosting situation?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 21 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Thu Jan 21 12:39:40 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 21 Jan 2010 12:39:40 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B583971.4060704@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B572088.7020501@holdenweb.com>
	<4B57542B.5060305@egenix.com>
	<94bdd2611001201334n5d7fffcckfcafdf25fe841895@mail.gmail.com>
	<4B577D2E.6060606@holdenweb.com>
	<94bdd2611001201413l53384babneb2433db37e844a5@mail.gmail.com>
	<4B578670.6010203@v.loewis.de>
	<94bdd2611001201505p839dc7arb4ec0aeae7b84194@mail.gmail.com>
	<4B57BC62.4090401@holdenweb.com> <4B583971.4060704@egenix.com>
Message-ID: <94bdd2611001210339v85b351dgc69fccf3665b1c9a@mail.gmail.com>

On Thu, Jan 21, 2010 at 12:24 PM, M.-A. Lemburg <mal at egenix.com> wrote:
[..]
>>>
>> I have in the past suggested that we consider hosting services at
>> diverse places. I'd have thought this was a prima facie case for
>> distributed hosting facilities. If we have that, we have no need for
>> mirrors, but instead for systems management. I know of at least three
>> reputable US hosting companies who I am pretty sure would help, and a
>> major academic hosting organization too. Also maybe Snakebite might be
>> another location?
>
> That would be my preference as well.
>
> Services like Amazon CloudFront or Akamai could easily scale up to
> millions of users.
>
> Since the PyPI data is already available as set of static files
> (at least from looking at the simple/ index), pushing this through
> such content delivery systems should easily be possible.
>
> Such a setup would avoid all the legalese issues, since the PSF
> would run the infrastructure and also make monitoring the service
> a lot easier.
>
> Moreover, all the complicated stuff like on-demand content
> distribution is handled by those services automatically.

I don't understand why we would bother maintaining such an
infrastructure, since the community itself is willing to maintain
mirrors.

As long as the mirror maintainers are willing to follow our policy,
it's just a matter of defining this policy I think.  And as Martin
said, we have the technical ability to make sure a mirror hasn't
changed
any content in the content mirrorred.

Regards
Tarek

From ziade.tarek at gmail.com  Thu Jan 21 12:42:51 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 21 Jan 2010 12:42:51 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B583398.9090509@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com> <4B572088.7020501@holdenweb.com>
	<4B5778D8.9090402@v.loewis.de> <4B577BA3.7070606@egenix.com>
	<4B577CE9.2050809@v.loewis.de> <4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>
Message-ID: <94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>

On Thu, Jan 21, 2010 at 11:59 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> "Martin v. L?wis" wrote:
>>> Sure, the PEP can be used as basis for the decision process, but
>>> someone still has to make the decision to add a mirror or not
>>> and these people should be appointed to by the PSF - much like we
>>> have an infrastructure committee to see after the python.org site.
>>>
>>> The situation is a lot like with the Python trademarks:
>>> The good guys always come and ask for permission. The bad guys
>>> don't.
>>
>> With PEP 381, the bad guys won't get any attention. You can already
>> run as many public mirrors of PyPI as you want, but nobody will notice.
>>
>> To become an official mirror, you have to ask for permission already;
>> see the PEP.
>>
>>> In order to go after them we need a clear set of
>>> rules for setting up and running a PyPI mirror and disallowing
>>> setups that don't follow these rules.
>>
>> See the PEP.
>
> Like I said: the PEP can be used to document the technical requirements
> of being accepted as official mirror, but it doesn't cover any
> of the legal requirements the PSF will need to put in place in
> order to prevent unofficial mirrors which use the content to
> e.g. increase their page rank, add advertisement and other
> drive-by revenue, misrepresent authorship, add malware to popular
> packages, etc.

As Martin mentioned yesterday, the latter can't happen : see
http://mail.python.org/pipermail/catalog-sig/2009-March/002018.html

Tarek

From mal at egenix.com  Thu Jan 21 12:51:44 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2010 12:51:44 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>
	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>
	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>
	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>
	<4B583398.9090509@egenix.com>
	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>
Message-ID: <4B583FD0.8000904@egenix.com>

Tarek Ziad? wrote:
> On Thu, Jan 21, 2010 at 11:59 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> "Martin v. L?wis" wrote:
>>>> Sure, the PEP can be used as basis for the decision process, but
>>>> someone still has to make the decision to add a mirror or not
>>>> and these people should be appointed to by the PSF - much like we
>>>> have an infrastructure committee to see after the python.org site.
>>>>
>>>> The situation is a lot like with the Python trademarks:
>>>> The good guys always come and ask for permission. The bad guys
>>>> don't.
>>>
>>> With PEP 381, the bad guys won't get any attention. You can already
>>> run as many public mirrors of PyPI as you want, but nobody will notice.
>>>
>>> To become an official mirror, you have to ask for permission already;
>>> see the PEP.
>>>
>>>> In order to go after them we need a clear set of
>>>> rules for setting up and running a PyPI mirror and disallowing
>>>> setups that don't follow these rules.
>>>
>>> See the PEP.
>>
>> Like I said: the PEP can be used to document the technical requirements
>> of being accepted as official mirror, but it doesn't cover any
>> of the legal requirements the PSF will need to put in place in
>> order to prevent unofficial mirrors which use the content to
>> e.g. increase their page rank, add advertisement and other
>> drive-by revenue, misrepresent authorship, add malware to popular
>> packages, etc.
> 
> As Martin mentioned yesterday, the latter can't happen : see
> http://mail.python.org/pipermail/catalog-sig/2009-March/002018.html

I'm not thinking of sites that make the data available to automated
tools like pip which could of course check signatures, etc.

The problem gets real when putting the data up on the web for
users to download via a browser. If they then install directly from
the file without checking signatures, they can easily be tricked
into executing malware - and that would put the original author
of such a package into a pretty bad light.

In any case, that was just a list of examples.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 21 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Thu Jan 21 13:06:19 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 21 Jan 2010 13:06:19 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B583FD0.8000904@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B572088.7020501@holdenweb.com>
	<4B5778D8.9090402@v.loewis.de> <4B577BA3.7070606@egenix.com>
	<4B577CE9.2050809@v.loewis.de> <4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>
	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>
	<4B583FD0.8000904@egenix.com>
Message-ID: <94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>

On Thu, Jan 21, 2010 at 12:51 PM, M.-A. Lemburg <mal at egenix.com> wrote:
[..]
> The problem gets real when putting the data up on the web for
> users to download via a browser. If they then install directly from
> the file without checking signatures, they can easily be tricked
> into executing malware - and that would put the original author
> of such a package into a pretty bad light.
>
> In any case, that was just a list of examples.

What about restricting the mirrors to the non web part in that case ?

Because the mirroring infrastructure is really intended for what I
would call a "professional" usage of PyPI, where it matters if it's
down for some time. And this usage is always done through automated
tools.

If the PyPI *website* part is down for a while, it's a minor annoyance
for people that are installing by clicking.

Then, in a second phase, we could have a second mirroring level with a
web part, and ask for the maintainer to sign a "mirror agreement" to
make him responsible in case he's a bad guy, and make him/her
acknowledge some PSF members maybe ? Because the people that are
willing to maintain mirrors are respected/known developers.

But the latter is not really what we need for our everyday work.

Regards,
Tarek

-- 
Tarek Ziad? | http://ziade.org

From mal at egenix.com  Thu Jan 21 17:29:17 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2010 17:29:17 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>
	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>
	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>
	<4B583398.9090509@egenix.com>	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>	<4B583FD0.8000904@egenix.com>
	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
Message-ID: <4B5880DD.8030509@egenix.com>

Tarek Ziad? wrote:
> On Thu, Jan 21, 2010 at 12:51 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> [..]
>> The problem gets real when putting the data up on the web for
>> users to download via a browser. If they then install directly from
>> the file without checking signatures, they can easily be tricked
>> into executing malware - and that would put the original author
>> of such a package into a pretty bad light.
>>
>> In any case, that was just a list of examples.
> 
> What about restricting the mirrors to the non web part in that case ?
> 
> Because the mirroring infrastructure is really intended for what I
> would call a "professional" usage of PyPI, where it matters if it's
> down for some time. And this usage is always done through automated
> tools.
> 
> If the PyPI *website* part is down for a while, it's a minor annoyance
> for people that are installing by clicking.
> 
> Then, in a second phase, we could have a second mirroring level with a
> web part, and ask for the maintainer to sign a "mirror agreement" to
> make him responsible in case he's a bad guy, and make him/her
> acknowledge some PSF members maybe ? Because the people that are
> willing to maintain mirrors are respected/known developers.
> 
> But the latter is not really what we need for our everyday work.

Sure, we could do all those things, but such a process will
cause a lot of admin overhead on part of the PSF.

Using a content delivery system we'd avoid such administration
work. The PSF would have to sign agreements with 10-20 mirror
providers, it wouldn't have to setup a monitoring system, keep
checking the mirror web content, etc.

Moreover, there would also be mirrors in parts of the world
that are currently not well covered by Pythonistas and thus
less likely to get a local mirror server setup.

How to arrange all this is really a PSF question more than
anything else.

Also note that using a static file layout would make the
whole synchronization mechanism a lot easier - not only
for content delivery networks, but also for dedicated
volunteer run mirrors. There are lots of mirror scripts
out there that work with rsync or FTP, so no need to reinvent
the wheel.

AFAICTL, all the data on PyPI is static and can be rendered
as files in a directory dump. A simple cronjob could take
care of this every few minutes or so and extract the data
to a local directory which is then made accessible to
mirrors.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 21 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Thu Jan 21 17:49:43 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 21 Jan 2010 17:49:43 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B5880DD.8030509@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B577BA3.7070606@egenix.com>
	<4B577CE9.2050809@v.loewis.de> <4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>
	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>
	<4B583FD0.8000904@egenix.com>
	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
	<4B5880DD.8030509@egenix.com>
Message-ID: <94bdd2611001210849x5ae885dbm555e5dfe1ac4dd2f@mail.gmail.com>

On Thu, Jan 21, 2010 at 5:29 PM, M.-A. Lemburg <mal at egenix.com> wrote:
[..]
>
> Sure, we could do all those things, but such a process will
> cause a lot of admin overhead on part of the PSF.

Which process ? the non-web mirroring requires no effort/work from the PSF.

The only effort that is required is technical, and 70% done at this
point I'd say.

> Using a content delivery system we'd avoid such administration
> work. The PSF would have to sign agreements with 10-20 mirror
> providers, it wouldn't have to setup a monitoring system, keep
> checking the mirror web content, etc.

What is a content delivery system here ? do you mean by that that the PSF
would run the mirror by itself ? if so, how this is going to work technically ?
how would it be different ?

Let me state it differently : what if each mirror maintainer is a PSF member ?
does that addresse the legal/admin issues ?

>
> Moreover, there would also be mirrors in parts of the world
> that are currently not well covered by Pythonistas and thus
> less likely to get a local mirror server setup.

This is just a matter of having a server IP in that part of the world.
And in reality, as long as the main areas US, Europe, Austrialia, etc
are served,
this fits our needs. But some people will probably have to go through
several nodes
to reach a mirror, but we can't have a server per major city.

So in any case, we are improving the situation, not making it worse.


> How to arrange all this is really a PSF question more than
> anything else.
>
> Also note that using a static file layout would make the
> whole synchronization mechanism a lot easier - not only
> for content delivery networks, but also for dedicated
> volunteer run mirrors. There are lots of mirror scripts
> out there that work with rsync or FTP, so no need to reinvent
> the wheel.
>

Those scripts already exist and are in usage in the tools that are
mirroring pypi.
They are not rsync but http calls, but that's about it.

> AFAICTL, all the data on PyPI is static and can be rendered
> as files in a directory dump. A simple cronjob could take
> care of this every few minutes or so and extract the data
> to a local directory which is then made accessible to
> mirrors.

People are already doing rsync-like mirrors. But that's quite an
incomplete mirror.

The whole point of the work I've been doing with Martin (partially
reflected in PEP 381)
is to be able to have the download statistics for each archive, no
matter which mirror was used to download the file.  That's quite a
valuable information.

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From mal at egenix.com  Thu Jan 21 20:08:18 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2010 20:08:18 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <94bdd2611001210849x5ae885dbm555e5dfe1ac4dd2f@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>
	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>
	<4B583398.9090509@egenix.com>	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>	<4B583FD0.8000904@egenix.com>	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>	<4B5880DD.8030509@egenix.com>
	<94bdd2611001210849x5ae885dbm555e5dfe1ac4dd2f@mail.gmail.com>
Message-ID: <4B58A622.6080301@egenix.com>

Tarek Ziad? wrote:
> On Thu, Jan 21, 2010 at 5:29 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> [..]
>>
>> Sure, we could do all those things, but such a process will
>> cause a lot of admin overhead on part of the PSF.
> 
> Which process ? the non-web mirroring requires no effort/work from the PSF.
> 
> The only effort that is required is technical, and 70% done at this
> point I'd say.

No, it's not only technical. The administration overhead
comes into play when looking at the legal side of things
and that does not only require doing initial checks of
whether the mirrors are adhering to a set of technical
standards, but also whether they adhere to legal ones.

We'd avoid all that with e.g. a cloud setup run by the
PSF and get all the monitoring, statistics, etc. for free.

>> Using a content delivery system we'd avoid such administration
>> work. The PSF would have to sign agreements with 10-20 mirror
>> providers, it wouldn't have to setup a monitoring system, keep
>> checking the mirror web content, etc.
> 
> What is a content delivery system here ? do you mean by that that the PSF
> would run the mirror by itself ? if so, how this is going to work technically ?
> how would it be different ?

See e.g. http://aws.amazon.com/cloudfront/ for such a system.

Using Amazon would even also the PSF to run the web front end
based on the same system and data.

The main advantage is that they take care of all the sys admin
stuff and provide a unified platform to work with.

> Let me state it differently : what if each mirror maintainer is a PSF member ?
> does that addresse the legal/admin issues ?

Perhaps the legal ones (can't say, we'd have to ask Van),
but certainly not the admin issues:

Each mirror maintainer would have to invest time in setting up
such a server, so instead of simplifying things, we'd make them
more work intense... and then you have to manage software updates
on those servers, fight network problems, handle differences
between server platforms and OSes, implement fail-over,
etc. etc. - basically all the usual operations stuff needed
to maintain a distributed cluster.

With a service like Amazon or Akamai you just have one platform
and/or API to worry about. All the sys admin overhead is done by
others and edge distribution comes right with the service.

>> Moreover, there would also be mirrors in parts of the world
>> that are currently not well covered by Pythonistas and thus
>> less likely to get a local mirror server setup.
> 
> This is just a matter of having a server IP in that part of the world.

You'd also have to have the data in that part of the world
if you want to benefit from a local mirror. That's what edge
distribution is all about.

> And in reality, as long as the main areas US, Europe, Austrialia, etc
> are served,
> this fits our needs. But some people will probably have to go through
> several nodes
> to reach a mirror, but we can't have a server per major city.
> 
> So in any case, we are improving the situation, not making it worse.

That's not what I'm saying. I hope I'm making myself clear.
If not please let me know.

What I'm trying to say is that it is more effective to look at
existing solutions to these standard problems and work from
there instead of reinventing yet another distribution network
mechanism from ground up.

This will save you a lot of work and it will also simplify
the legalese and administration from the PSF side of things.

>> How to arrange all this is really a PSF question more than
>> anything else.
>>
>> Also note that using a static file layout would make the
>> whole synchronization mechanism a lot easier - not only
>> for content delivery networks, but also for dedicated
>> volunteer run mirrors. There are lots of mirror scripts
>> out there that work with rsync or FTP, so no need to reinvent
>> the wheel.
>>
> 
> Those scripts already exist and are in usage in the tools that are
> mirroring pypi. They are not rsync but http calls, but that's about it.

Ok, so that wheel has already been reinvented :-)

>> AFAICTL, all the data on PyPI is static and can be rendered
>> as files in a directory dump. A simple cronjob could take
>> care of this every few minutes or so and extract the data
>> to a local directory which is then made accessible to
>> mirrors.
> 
> People are already doing rsync-like mirrors. But that's quite an
> incomplete mirror.

Why is that ? Why can't the PyPI data be extracted to the
file system to make it more accessible to standard tools ?

> The whole point of the work I've been doing with Martin (partially
> reflected in PEP 381)
> is to be able to have the download statistics for each archive, no
> matter which mirror was used to download the file.  That's quite a
> valuable information.

Indeed and Amazon will provide those to you without having
to do any extra work:

http://aws.amazon.com/about-aws/whats-new/2009/05/07/amazon-cloudfront-adds-access-logging-capability/
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2440

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 21 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Thu Jan 21 21:17:14 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 21 Jan 2010 21:17:14 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B58A622.6080301@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>
	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>
	<4B583FD0.8000904@egenix.com>
	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
	<4B5880DD.8030509@egenix.com>
	<94bdd2611001210849x5ae885dbm555e5dfe1ac4dd2f@mail.gmail.com>
	<4B58A622.6080301@egenix.com>
Message-ID: <94bdd2611001211217i1624f130k1ee465b91f08f1c4@mail.gmail.com>

On Thu, Jan 21, 2010 at 8:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
[..]
>> Those scripts already exist and are in usage in the tools that are
>> mirroring pypi. They are not rsync but http calls, but that's about it.
>
> Ok, so that wheel has already been reinvented :-)

That's historical. The HTTP interface existed already and is used by
PIP and al. People built mirroring scripts on the top of it, and used
the changelog file to get the updates.

IOW, this works *today* with the existing software.

Anyways, I guess I'll wait for the PSF decision to continue on that PEP.

Last, if the PSF-maintained cloud-based system is preferred over what
we've worked on,
it would be great if it's still built on the top of an open protocol
that is not tied to a cloud system, because people will be able to use
it for their own PyPI systems.

For instance, plone.org has a PyPI interface nowadays, so people can
upload distributions over there using Distutils. And it could have
mirrors as well. And accurate stats... :)

Regards,
Tarek

-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Thu Jan 21 21:57:00 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 21 Jan 2010 21:57:00 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B583398.9090509@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>
	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>
Message-ID: <4B58BF9C.6060403@v.loewis.de>

> Like I said: the PEP can be used to document the technical requirements
> of being accepted as official mirror, but it doesn't cover any
> of the legal requirements the PSF will need to put in place in
> order to prevent unofficial mirrors

Ok - as we are discussing official only mirrors, why are you now
changing the subject to unofficial mirrors?

Regards,
Martin


From martin at v.loewis.de  Thu Jan 21 22:06:37 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 21 Jan 2010 22:06:37 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI
	usage	agreement
In-Reply-To: <94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>
	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>
	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>
	<4B583398.9090509@egenix.com>	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>	<4B583FD0.8000904@egenix.com>
	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
Message-ID: <4B58C1DD.6040307@v.loewis.de>

> What about restricting the mirrors to the non web part in that case ?

I think MAL is talking about a completely different setup: the
unofficial mirror. The unofficial mirror doesn't follow any protocol;
it's just a mirror of PyPI using the standard API to fetch all data
available. People have been operating unofficial mirrors of various
qualities for several years.

Now, MAL is then concerned about malicious users of unofficial servers.
They might try to get their mirror high-ranking in Google, and then
redirect regular PyPI users to them. There isn't anything that the PEP
can do about that.

There has only been one such malicious mirror in the past. The PSF legal
council was fairly helpful in dealing with that case.

Regards,
Martin

From ianb at colorstudy.com  Thu Jan 21 22:49:14 2010
From: ianb at colorstudy.com (Ian Bicking)
Date: Thu, 21 Jan 2010 15:49:14 -0600
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <20100120005756.8D9B13A4065@sparrow.telecommunity.com>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com> 
	<20100119210547.08AEE3A4065@sparrow.telecommunity.com>
	<4B562255.2050709@v.loewis.de> 
	<20100120005756.8D9B13A4065@sparrow.telecommunity.com>
Message-ID: <b654cd2e1001211349u3ee8f5b6h8b094ee20a1680c4@mail.gmail.com>

If you use <base href> that might resolve this problem (pip looks at this, I
haven't checked the setuptools code).

pip does not use the Setuptools index code, and I don't have unescaping in
pip, so the implementations differ in this case.  I guess I didn't notice
because in tests I didn't encounter any (meaningful) links with &amp; in
them.

On Tue, Jan 19, 2010 at 6:57 PM, P.J. Eby <pje at telecommunity.com> wrote:

> At 10:21 PM 1/19/2010 +0100, Martin v. L?wis wrote:
>
>> >> 2. Make file paths relative (from
>> >> <http://pypi.python.org/packages>http://pypi.python.org/packages
>> >> ?  to ../../packages)
>> ...
>> > Likewise, #2 should be fine as long as urlparse.urljoin produces the
>> > correct result using the retrieved-from URL as the base URL.  I stress
>> > this because some web applications treat the urls "foo" and "foo/" as if
>> > they were interchangeable, and for relative-URL purposes, they are not.
>> > So, as long as PyPI handles this correctly based on the URL that was
>> > sent by the *client* (rather than perhaps a canonical URL the client
>> > "should have" sent), then that will be fine.
>>
>> Can you please suggest an example where this could break? If the client
>> doesn't send an absolute path, it will be incorrect HTTP, no?
>>
>
> No, what I mean is that if PyPI returns *identical* HTML for requests going
> to "/foo" and "/foo/", then one of those pages will contain incorrect
> relative URLs.
>
> I mention this because, at least in the main web interface, going to
> http://pypi.python.org/pypi/setuptools and
> http://pypi.python.org/pypi/setuptools/ produces similar (if not
> identical) HTML.
>
> Currently, the relative URLs in such pages are site-absolute, i.e.,
> "/pypi/whatever", so the fact that the two pages are identical isn't a
> problem.  However, if you used pure relative URLs ('../whatever'), then one
> of the two pages would have incorrect relative URLs.
>
> So, all I am saying is, please make sure that the generated relative URLs
> are correct, based not on a canonical form of the URL, but based on whether
> the client asked for http://pypi.python.org/simple/setuptools or
> http://pypi.python.org/simple/setuptools/ - because if those two URLs
> return *identical* HTML, *one* of them will be wrong.
>
> In other words, one of those two pages must have one more '../' in each URL
> than the other one has for the same URL.  Does that make sense now?
>
> I'm really just saying that the relative URLs must be correct, and
> reminding you (in case you're not already aware) that the current PyPI code
> may be relying on its use of "/pypi" relative URLs in order to ignore the
> trailing-slash issue.  And if that's the case, then there will be pages
> generated with *wrong* relative URLs, unless you remember to test both the
> '/'-terminated and non-'/'-terminated versions.  (Even if just by clicking
> the links in a browser.)
>
>


-- 
Ian Bicking  |  http://blog.ianbicking.org  |
http://topplabs.org/civichacker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100121/7c505781/attachment.htm>

From ziade.tarek at gmail.com  Thu Jan 21 22:54:33 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 21 Jan 2010 22:54:33 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B58C1DD.6040307@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B577BA3.7070606@egenix.com>
	<4B577CE9.2050809@v.loewis.de> <4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>
	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>
	<4B583FD0.8000904@egenix.com>
	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
	<4B58C1DD.6040307@v.loewis.de>
Message-ID: <94bdd2611001211354r3508789eg23172f00bd9ea272@mail.gmail.com>

2010/1/21 "Martin v. L?wis" <martin at v.loewis.de>:
>> What about restricting the mirrors to the non web part in that case ?
>
> I think MAL is talking about a completely different setup: the
> unofficial mirror. The unofficial mirror doesn't follow any protocol;
> it's just a mirror of PyPI using the standard API to fetch all data
> available. People have been operating unofficial mirrors of various
> qualities for several years.

OK. That's not what I understood since he proposed a cloud system run
by the PSF for the PyPI mirrors.  Which implied (to me) those were
official mirrors.

The goal of the PEP is to address the official mirrors of course,
listed at the mirrors.pypi.python.org DNS (and also provide a protocol
others can reuse as well)

Tarek

-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Thu Jan 21 23:15:08 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Thu, 21 Jan 2010 23:15:08 +0100
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <b654cd2e1001211349u3ee8f5b6h8b094ee20a1680c4@mail.gmail.com>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
	<20100119210547.08AEE3A4065@sparrow.telecommunity.com>
	<4B562255.2050709@v.loewis.de>
	<20100120005756.8D9B13A4065@sparrow.telecommunity.com>
	<b654cd2e1001211349u3ee8f5b6h8b094ee20a1680c4@mail.gmail.com>
Message-ID: <4B58D1EC.8010001@v.loewis.de>

> If you use <base href> that might resolve this problem (pip looks at
> this, I haven't checked the setuptools code).

Thanks for the hint - unfortunately, that would defeat the purpose;
if the pages are mirrored as-is, they would then refer back to
python.org, whereas having them truly relative was exactly desired
for the mirroring.

So I guess I'll just arrange for the pages to depend on the
exact URL being queried. According to the setuptools spec, the
version with the trailing slash is the "official" one.

> pip does not use the Setuptools index code, and I don't have unescaping
> in pip, so the implementations differ in this case.  I guess I didn't
> notice because in tests I didn't encounter any (meaningful) links with
> &amp; in them.

So I'll go ahead and do the escaping. If this matters in practice,
somebody will surely complain.

Regards,
Martin

From martin at v.loewis.de  Thu Jan 21 23:19:53 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 21 Jan 2010 23:19:53 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <94bdd2611001211354r3508789eg23172f00bd9ea272@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B577BA3.7070606@egenix.com>	
	<4B577CE9.2050809@v.loewis.de> <4B57807F.2050303@egenix.com>	
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>	
	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>	
	<4B583FD0.8000904@egenix.com>	
	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>	
	<4B58C1DD.6040307@v.loewis.de>
	<94bdd2611001211354r3508789eg23172f00bd9ea272@mail.gmail.com>
Message-ID: <4B58D309.8000704@v.loewis.de>

> OK. That's not what I understood since he proposed a cloud system run
> by the PSF for the PyPI mirrors.  Which implied (to me) those were
> official mirrors.

For some reason (which I don't understand) MAL is opposed to the notion
of mirrors. If the complaints about a mirroring system are somehow
fundamental, there isn't anything we can do about it, so I propose we
just proceed, anyway. The way the PEP process is structured, it would
then be up to the BDFL (not the PSF board) to ultimately decide on the
entire subject.

Regards,
Martin

From pje at telecommunity.com  Fri Jan 22 00:07:06 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 21 Jan 2010 18:07:06 -0500
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <4B58D1EC.8010001@v.loewis.de>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
	<20100119210547.08AEE3A4065@sparrow.telecommunity.com>
	<4B562255.2050709@v.loewis.de>
	<20100120005756.8D9B13A4065@sparrow.telecommunity.com>
	<b654cd2e1001211349u3ee8f5b6h8b094ee20a1680c4@mail.gmail.com>
	<4B58D1EC.8010001@v.loewis.de>
Message-ID: <20100121230718.1AC3C3A4065@sparrow.telecommunity.com>

At 11:15 PM 1/21/2010 +0100, Martin v. L??wis wrote:
> > If you use <base href> that might resolve this problem (pip looks at
> > this, I haven't checked the setuptools code).
>
>Thanks for the hint - unfortunately, that would defeat the purpose;
>if the pages are mirrored as-is, they would then refer back to
>python.org, whereas having them truly relative was exactly desired
>for the mirroring.
>
>So I guess I'll just arrange for the pages to depend on the
>exact URL being queried. According to the setuptools spec, the
>version with the trailing slash is the "official" one.

A simple fix may be to make the unterminated URLs issue a redirect to 
the /-terminated version, i.e. make /foo redirect to /foo/ - this is 
what both Apache and Zope do for such cases, and setuptools uses the 
final resolved URL as its base URL.

Anyway, that would allow you to not worry about generating correct 
URLs for the unterminated case.


From david.lyon at pythontest.org  Fri Jan 22 02:08:57 2010
From: david.lyon at pythontest.org (David Lyon)
Date: Fri, 22 Jan 2010 12:08:57 +1100 (EST)
Subject: [Catalog-sig] Package Testing on pypi - revisited
In-Reply-To: <94bdd2611001210849x5ae885dbm555e5dfe1ac4dd2f@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au> <4B577BA3.7070606@egenix.com>
	<4B577CE9.2050809@v.loewis.de> <4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>
	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>
	<4B583FD0.8000904@egenix.com>
	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
	<4B5880DD.8030509@egenix.com>
	<94bdd2611001210849x5ae885dbm555e5dfe1ac4dd2f@mail.gmail.com>
Message-ID: <36010.115.128.25.96.1264122537.squirrel@syd-srv02.ezyreg.com>

Hi Tarek,

> The whole point of the work I've been doing with Martin (partially
> reflected in PEP 381)
> is to be able to have the download statistics for each archive, no
> matter which mirror was used to download the file.  That's quite a
> valuable information.

Perphaps it is valuable.

but what is your take on package metrics information and package
testing ?

Some years ago, well before my time, there was cpants.org. It did
testing of package quality.

That information never seems to have never made it on to the pages
of pypi. That would be *much more* valuable to have.

I've been looking at the cheesecake code and can't half admire all
of the effort that has gone into that package. It would be such
a shame to throw that effort away.

Are we actually abandoning or don't want package testing ?

My take is that the output report could be corrected to take out
offensive english ("kwalitee") and a graph produced to give some
very quick sort of rating.

The whole thing could exist as a 300x200 pixel bitmap perphaps.

I'd be happy to do the work, but need to get your feedback first.

Regards

David






From ianb at colorstudy.com  Fri Jan 22 07:35:21 2010
From: ianb at colorstudy.com (Ian Bicking)
Date: Fri, 22 Jan 2010 00:35:21 -0600
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <20100121230718.1AC3C3A4065@sparrow.telecommunity.com>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com> 
	<20100119210547.08AEE3A4065@sparrow.telecommunity.com>
	<4B562255.2050709@v.loewis.de> 
	<20100120005756.8D9B13A4065@sparrow.telecommunity.com>
	<b654cd2e1001211349u3ee8f5b6h8b094ee20a1680c4@mail.gmail.com> 
	<4B58D1EC.8010001@v.loewis.de>
	<20100121230718.1AC3C3A4065@sparrow.telecommunity.com>
Message-ID: <b654cd2e1001212235t3424ef57ub2b59fd785bb473a@mail.gmail.com>

On Thu, Jan 21, 2010 at 5:07 PM, P.J. Eby <pje at telecommunity.com> wrote:

> At 11:15 PM 1/21/2010 +0100, Martin v. L?wis wrote:
>
>> > If you use <base href> that might resolve this problem (pip looks at
>> > this, I haven't checked the setuptools code).
>>
>> Thanks for the hint - unfortunately, that would defeat the purpose;
>> if the pages are mirrored as-is, they would then refer back to
>> python.org, whereas having them truly relative was exactly desired
>> for the mirroring.
>>
>> So I guess I'll just arrange for the pages to depend on the
>> exact URL being queried. According to the setuptools spec, the
>> version with the trailing slash is the "official" one.
>>
>
> A simple fix may be to make the unterminated URLs issue a redirect to the
> /-terminated version, i.e. make /foo redirect to /foo/ - this is what both
> Apache and Zope do for such cases, and setuptools uses the final resolved
> URL as its base URL.
>
> Anyway, that would allow you to not worry about generating correct URLs for
> the unterminated case.
>

Agreed, this is the best solution generally to ambiguous /'s.  Though
sometimes it's easier to always strip /'s; you can do that globally, but you
can't add /'s globally.

-- 
Ian Bicking  |  http://blog.ianbicking.org  |
http://topplabs.org/civichacker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100122/d32341e9/attachment.htm>

From pje at telecommunity.com  Fri Jan 22 07:42:44 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 22 Jan 2010 01:42:44 -0500
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <b654cd2e1001212235t3424ef57ub2b59fd785bb473a@mail.gmail.co
 m>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
	<20100119210547.08AEE3A4065@sparrow.telecommunity.com>
	<4B562255.2050709@v.loewis.de>
	<20100120005756.8D9B13A4065@sparrow.telecommunity.com>
	<b654cd2e1001211349u3ee8f5b6h8b094ee20a1680c4@mail.gmail.com>
	<4B58D1EC.8010001@v.loewis.de>
	<20100121230718.1AC3C3A4065@sparrow.telecommunity.com>
	<b654cd2e1001212235t3424ef57ub2b59fd785bb473a@mail.gmail.com>
Message-ID: <20100122064252.265C33A4065@sparrow.telecommunity.com>

At 12:35 AM 1/22/2010 -0600, Ian Bicking wrote:
>Agreed, this is the best solution generally to ambiguous /'s. ? 
>Though sometimes it's easier to always strip /'s; you can do that 
>globally, but you can't add /'s globally.

I don't follow you.  Without the redirect, you can't get the client 
to use a different base URL, whether you're adding or removing the 
trailing slash, or doing any other change whatsoever AFAIK.


From mal at egenix.com  Fri Jan 22 10:43:01 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 22 Jan 2010 10:43:01 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <94bdd2611001211217i1624f130k1ee465b91f08f1c4@mail.gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>
	<4B583398.9090509@egenix.com>	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>	<4B583FD0.8000904@egenix.com>	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>	<4B5880DD.8030509@egenix.com>	<94bdd2611001210849x5ae885dbm555e5dfe1ac4dd2f@mail.gmail.com>	<4B58A622.6080301@egenix.com>
	<94bdd2611001211217i1624f130k1ee465b91f08f1c4@mail.gmail.com>
Message-ID: <4B597325.3070906@egenix.com>

Tarek Ziad? wrote:
> On Thu, Jan 21, 2010 at 8:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> [..]
>>> Those scripts already exist and are in usage in the tools that are
>>> mirroring pypi. They are not rsync but http calls, but that's about it.
>>
>> Ok, so that wheel has already been reinvented :-)
> 
> That's historical. The HTTP interface existed already and is used by
> PIP and al. People built mirroring scripts on the top of it, and used
> the changelog file to get the updates.
> 
> IOW, this works *today* with the existing software.
> 
> Anyways, I guess I'll wait for the PSF decision to continue on that PEP.
> 
> Last, if the PSF-maintained cloud-based system is preferred over what
> we've worked on,
> it would be great if it's still built on the top of an open protocol
> that is not tied to a cloud system, because people will be able to use
> it for their own PyPI systems.
> 
> For instance, plone.org has a PyPI interface nowadays, so people can
> upload distributions over there using Distutils. And it could have
> mirrors as well. And accurate stats... :)

Sure :-)

Sorry for spoiling the fun a bit, but like you said: this is a PSF
decision to make and has more strings attached to it than "just"
technical ones.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 22 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Fri Jan 22 10:47:54 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 22 Jan 2010 10:47:54 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B58BF9C.6060403@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>
	<4B583398.9090509@egenix.com> <4B58BF9C.6060403@v.loewis.de>
Message-ID: <4B59744A.7040103@egenix.com>

"Martin v. L?wis" wrote:
>> Like I said: the PEP can be used to document the technical requirements
>> of being accepted as official mirror, but it doesn't cover any
>> of the legal requirements the PSF will need to put in place in
>> order to prevent unofficial mirrors
> 
> Ok - as we are discussing official only mirrors, why are you now
> changing the subject to unofficial mirrors?

Please see my original email: the good guys will always try to
play nice, the bad guys won't.

In order to make it clear that PyPI data may only be mirrored
for redistribution with PSF authorization, we need to add proper
notices to PyPI and also prevent such mirroring technically
(if possible).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 22 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Fri Jan 22 10:52:08 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 22 Jan 2010 10:52:08 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to
	PyPI	usage	agreement
In-Reply-To: <4B58C1DD.6040307@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>	<4B583FD0.8000904@egenix.com>	<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>
	<4B58C1DD.6040307@v.loewis.de>
Message-ID: <4B597548.5050703@egenix.com>

"Martin v. L?wis" wrote:
>> What about restricting the mirrors to the non web part in that case ?
> 
> I think MAL is talking about a completely different setup: the
> unofficial mirror. The unofficial mirror doesn't follow any protocol;
> it's just a mirror of PyPI using the standard API to fetch all data
> available. People have been operating unofficial mirrors of various
> qualities for several years.
> 
> Now, MAL is then concerned about malicious users of unofficial servers.
> They might try to get their mirror high-ranking in Google, and then
> redirect regular PyPI users to them. There isn't anything that the PEP
> can do about that.

Right.

> There has only been one such malicious mirror in the past. The PSF legal
> council was fairly helpful in dealing with that case.

Indeed, but only on trademark terms, not based on rules that
cover permission to mirror the data in the first place.

All that said, I think we can get a much simpler setup working
using an existing cloud distribution system. This will solve the
legal issues, the technical ones, simplify monitoring, remove the
need to have sys admins looking after the mirror servers, provide
statistics, etc. etc.

And it's cost effective too, since the time spent on all of the
above values a lot higher than the few dollars it costs to run
such a setup on Akamai or Amazon.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 22 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Fri Jan 22 10:58:51 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 22 Jan 2010 10:58:51 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
 agreement
In-Reply-To: <4B58D309.8000704@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<4B577BA3.7070606@egenix.com>		<4B577CE9.2050809@v.loewis.de>
	<4B57807F.2050303@egenix.com>		<4B5786ED.2020708@v.loewis.de>
	<4B583398.9090509@egenix.com>		<94bdd2611001210342k411d1ad1la4ec28f1d2f57756@mail.gmail.com>		<4B583FD0.8000904@egenix.com>		<94bdd2611001210406w16c90986gaef8d1b684840bf5@mail.gmail.com>		<4B58C1DD.6040307@v.loewis.de>	<94bdd2611001211354r3508789eg23172f00bd9ea272@mail.gmail.com>
	<4B58D309.8000704@v.loewis.de>
Message-ID: <4B5976DB.6040700@egenix.com>

"Martin v. L?wis" wrote:
>> OK. That's not what I understood since he proposed a cloud system run
>> by the PSF for the PyPI mirrors.  Which implied (to me) those were
>> official mirrors.
> 
> For some reason (which I don't understand) MAL is opposed to the notion
> of mirrors. If the complaints about a mirroring system are somehow
> fundamental, there isn't anything we can do about it, so I propose we
> just proceed, anyway. The way the PEP process is structured, it would
> then be up to the BDFL (not the PSF board) to ultimately decide on the
> entire subject.

The PEP only covers the technical issues. I'm not arguing against
anything technical here.

When it comes to the implementation of the PEP on python.org,
things look different though. The PSF is running the python.org site
and PyPI. It is also the legal partner when signing up for PyPI usage
and needs to looks after that uploaded data.

As a result, the PSF has to decide what to do about mirrors.

For plone.org's PyPI installation, the Plone Foundation would have to
decide, for other PyPI installation the resp. entities running those
installations will have to decide.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 22 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ssteinerx at gmail.com  Fri Jan 22 18:26:38 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Fri, 22 Jan 2010 12:26:38 -0500
Subject: [Catalog-sig] Fwd: [XML-SIG] Something's definitely wrong...
References: <1C322D80-60C9-4FC1-86EA-B2F83B2FC366@gmail.com>
Message-ID: <E67129DB-E019-4C84-BB73-E2EEAEE3BDD1@gmail.com>

This is copied from the XML-SIG which doesn't seem to be too heavily followed...

I'm working on finishing up my pypi metadata mirror and have run into an apparent XML-RPC MultiCall issue.

This is using the MultiCall RPC interface on PyPi with my currently unversioned "pypimirror" user-agent (in case someone who as access could look in the PyPI logs for the bombing call).

Anyone used this interface on PyPI and/or have any ideas on this?

Thanks,

S

>> From: "Dieter Maurer" <dieter at handshake.de>
>> Date: January 21, 2010 8:42:09 AM EST
>> To: "ssteiner at idc" <ssteiner at integrateddevcorp.com>
>> Cc: xml-sig at python.org
>> Subject: Re: [XML-SIG] Something's definitely wrong...
>> 
>> ssteiner at idc wrote at 2010-1-20 08:43 -0500:
>>> I'm using the xmlrpc MultiCall class pretty heavily in an application and every time I've somehow caused an error in the xmlrpclib.py code, I get an exception trying to raise the exception.
>>> 
>>> Here's the most recent example: 
>>> 
>>> grouped = grouper(2, tuple(mc_result))
>>> File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/xmlrpclib.py", line 1001, in __getitem__
>>> raise Fault(item['faultCode'], item['faultString'])
>>> Fault: <Fault 1: "<type 'exceptions.TypeError'>:'NoneType' object is unsubscriptable">
>> 
>> You see here the client side code to report that an exception
>> has happened on the server side.
>> 
>> The "TypeError" was not raised at this place but on the server side.
>> Here, a "Fault" is (successfully) raised with "faultCode" 1 and "faultString"
>> "<type 'exceptions.TypeError'>:'NoneType' object is unsubscriptable">.
>> 
>> If you are lucky, the server side has logged information for
>> the raised exception. If not, you need to convince the server side
>> to do so.
>> 
>> -- 
>> Dieter
>> _______________________________________________
>> XML-SIG maillist  -  XML-SIG at python.org
>> http://mail.python.org/mailman/listinfo/xml-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100122/494f31ae/attachment.htm>

From martin at v.loewis.de  Fri Jan 22 23:11:09 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 22 Jan 2010 23:11:09 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B59744A.7040103@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>
	<4B583398.9090509@egenix.com> <4B58BF9C.6060403@v.loewis.de>
	<4B59744A.7040103@egenix.com>
Message-ID: <4B5A227D.4040307@v.loewis.de>

> In order to make it clear that PyPI data may only be mirrored
> for redistribution with PSF authorization, we need to add proper
> notices to PyPI and also prevent such mirroring technically
> (if possible).

However, I don't think this is factually the case: *anybody*
can indeed mirror the data in any way they like. This is how
it is, and how it should be.

Regards,
Martin


From pje at telecommunity.com  Sat Jan 23 00:44:52 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 22 Jan 2010 18:44:52 -0500
Subject: [Catalog-sig]
 [PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5A227D.4040307@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>
	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>
	<4B193350.7090609@python.org> <4B1961D1.7010405@egenix.com>
	<019d01ca7518$2dba00f0$892e02d0$@net> <4B1D5CCB.4000309@egenix.com>
	<4B1D61C3.7030705@python.org> <4B1D7601.5040608@egenix.com>
	<4B1DB9BE.5040000@python.org> <4B1FE7B1.9030608@egenix.com>
	<4B54D896.2060108@egenix.com> <4B553366.2090809@holdenweb.com>
	<4B56D0C0.2080703@egenix.com> <4B572088.7020501@holdenweb.com>
	<4B5778D8.9090402@v.loewis.de> <4B577BA3.7070606@egenix.com>
	<4B577CE9.2050809@v.loewis.de> <4B57807F.2050303@egenix.com>
	<4B5786ED.2020708@v.loewis.de> <4B583398.9090509@egenix.com>
	<4B58BF9C.6060403@v.loewis.de> <4B59744A.7040103@egenix.com>
	<4B5A227D.4040307@v.loewis.de>
Message-ID: <20100122234501.610B63A4075@sparrow.telecommunity.com>

At 11:11 PM 1/22/2010 +0100, Martin v. L?wis wrote:
> > In order to make it clear that PyPI data may only be mirrored
> > for redistribution with PSF authorization, we need to add proper
> > notices to PyPI and also prevent such mirroring technically
> > (if possible).
>
>However, I don't think this is factually the case: *anybody*
>can indeed mirror the data in any way they like. This is how
>it is, and how it should be.

Indeed.  And if we are talking about mirroring the contents of the 
/simple index, there is no intellectual property there for PSF to 
claim copyright on.  (At least in the U.S., where it's well 
established that you can't copyright a list of names and phone 
numbers... and a list of links that isn't creatively arranged or 
generated by a human editor is very likely in the same category.)

That doesn't mean you can't use clickwrap agreements and such, but 
really, trademark dilution and tarnishment are the kinds of laws that 
are specifically designed to address the kinds of problems that 
"evil" mirrors would cause.  Adding an "Official PyPI Content" mark 
to the PSF's trademarks and establishing usage requirements around 
the mark is probably a more fruitful path than seeking technical 
solutions to an essentially social/legal problem.


From mal at egenix.com  Sat Jan 23 10:40:29 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 23 Jan 2010 10:40:29 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5A227D.4040307@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>
	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>
	<4B5A227D.4040307@v.loewis.de>
Message-ID: <4B5AC40D.4030808@egenix.com>

"Martin v. L?wis" wrote:
>> In order to make it clear that PyPI data may only be mirrored
>> for redistribution with PSF authorization, we need to add proper
>> notices to PyPI and also prevent such mirroring technically
>> (if possible).
> 
> However, I don't think this is factually the case: *anybody*
> can indeed mirror the data in any way they like. This is how
> it is, and how it should be.

Sure, downloading things from PyPI is its main intent and
that's also what the proposed PyPI terms allow the PSF to
do.

However, there's a huge difference between just downloading
content and redistributing it. The latter is what we're discussing
here.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 23 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From martin at v.loewis.de  Sat Jan 23 11:03:42 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 23 Jan 2010 11:03:42 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5AC40D.4030808@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>
	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>
	<4B5A227D.4040307@v.loewis.de> <4B5AC40D.4030808@egenix.com>
Message-ID: <4B5AC97E.9010309@v.loewis.de>

>> However, I don't think this is factually the case: *anybody*
>> can indeed mirror the data in any way they like. This is how
>> it is, and how it should be.
> 
> Sure, downloading things from PyPI is its main intent and
> that's also what the proposed PyPI terms allow the PSF to
> do.
> 
> However, there's a huge difference between just downloading
> content and redistributing it. The latter is what we're discussing
> here.

Indeed, that's what we are discussing here. And, I can only
repeat myself: anybody can download the data and redistribute
it in any way they like. This is how it is, and how it should
be.

Regards,
Martin


From mal at egenix.com  Sat Jan 23 13:49:25 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 23 Jan 2010 13:49:25 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5AC97E.9010309@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>
	<4B5AC40D.4030808@egenix.com> <4B5AC97E.9010309@v.loewis.de>
Message-ID: <4B5AF055.3080505@egenix.com>

"Martin v. L?wis" wrote:
>>> However, I don't think this is factually the case: *anybody*
>>> can indeed mirror the data in any way they like. This is how
>>> it is, and how it should be.
>>
>> Sure, downloading things from PyPI is its main intent and
>> that's also what the proposed PyPI terms allow the PSF to
>> do.
>>
>> However, there's a huge difference between just downloading
>> content and redistributing it. The latter is what we're discussing
>> here.
> 
> Indeed, that's what we are discussing here. And, I can only
> repeat myself: anybody can download the data and redistribute
> it in any way they like. This is how it is, and how it should
> be.

They can technically, yes, but the fact that they can doesn't
map to any kind of permission for doing so.

Otherwise, anyone could take any content on the web and copy it
freely and use it for their own purposes, without having to ask
for permission or being restricted in the way the content is
used.

The package authors who signed up before the change to the PyPI
registration terms on 2009-11-29 have not given this permission:

"""
By signing up to this system, you agree to:
 * Not upload inappropriate material (only Python packages are
   allowed).
 * You agree that all information you post is published (email
   addresses are obfuscated).
"""

so being ignorant about this doesn't really help.

The authors who have signed up since were forced to accept
terms which do give permission to anyone to do anything
they like with the content - and that's what started this
thread: the terms are too permissive to be acceptable.

"""
By registering to upload content to PyPI, I agree and affirmatively acknowledge the following:

   1. Content is restricted to Python packages and related information only.
   2. Any content uploaded to PyPI is provided on a non-confidential basis.
   3. The PSF is free to use or disseminate any content that I upload on an unrestricted basis for
any purpose. In particular, the PSF and all other users of the web site are granted an irrevocable,
worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform,
and publish the content, including in digital form.
   4. I represent and warrant that I have complied with all government regulations concerning the
transfer or export of any content I upload to PyPI. In particular, if I am subject to United States
law, I represent and warrant that I have obtained the proper governmental authorization for the
export of the content I upload. I further affirm that any content I provide is not intended for use
by a government end-user as defined in part 772 of the United States Export Administration Regulations.
"""

The version I proposed restricts those permissions to redistribution by the
PSF - which is enough to run PyPI services:

"""
PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to
PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following:

1. Content is restricted to Python packages and related information only.

2. Any content uploaded to PyPI is provided on a non-confidential basis.

3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce,
distribute, transmit, display, perform, and publish the content, including in digital form. This
licence is for the sole purpose of enabling the PSF to display, distribute and promote the content
on PyPI.

4. I represent and warrant that I have complied with all government regulations concerning the
transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if
I am subject to United States law, I represent and warrant that I have obtained the proper
governmental authorization for the export of the content I upload. I further affirm that any content
I provide is not intended for use by a government end-user as defined in part 772 of the United
States Export Administration Regulations.
"""

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 23 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From martin at v.loewis.de  Sat Jan 23 14:05:08 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 23 Jan 2010 14:05:08 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5AF055.3080505@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>
	<4B5AC40D.4030808@egenix.com> <4B5AC97E.9010309@v.loewis.de>
	<4B5AF055.3080505@egenix.com>
Message-ID: <4B5AF404.3080908@v.loewis.de>

> They can technically, yes, but the fact that they can doesn't
> map to any kind of permission for doing so.

Most certainly it does: everybody is permitted to create mirrors.

> Otherwise, anyone could take any content on the web and copy it
> freely and use it for their own purposes, without having to ask
> for permission or being restricted in the way the content is
> used.

And indeed, everybody can.

> The authors who have signed up since were forced to accept
> terms which do give permission to anyone to do anything
> they like with the content - and that's what started this
> thread: the terms are too permissive to be acceptable.

No, they are not too permissive - they are exactly right.

> The version I proposed restricts those permissions to redistribution by the
> PSF - which is enough to run PyPI services:

If the legal council of the PSF asks me to change anything, I will.
However, I fail to see the point of making such a change. Users *have*
to have permission to make copies of the PyPI content, else PyPI would
be completely pointless. If users don't want stuff they upload to be
copied, they shouldn't upload it. It has always been that way.

Regards,
Martin

From steve at holdenweb.com  Sat Jan 23 14:14:06 2010
From: steve at holdenweb.com (Steve Holden)
Date: Sat, 23 Jan 2010 08:14:06 -0500
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5AF055.3080505@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>
	<4B5AC40D.4030808@egenix.com> <4B5AC97E.9010309@v.loewis.de>
	<4B5AF055.3080505@egenix.com>
Message-ID: <4B5AF61E.10004@holdenweb.com>

M.-A. Lemburg wrote:
> "Martin v. L?wis" wrote:
>>>> However, I don't think this is factually the case: *anybody*
>>>> can indeed mirror the data in any way they like. This is how
>>>> it is, and how it should be.
>>> Sure, downloading things from PyPI is its main intent and
>>> that's also what the proposed PyPI terms allow the PSF to
>>> do.
>>>
>>> However, there's a huge difference between just downloading
>>> content and redistributing it. The latter is what we're discussing
>>> here.
>> Indeed, that's what we are discussing here. And, I can only
>> repeat myself: anybody can download the data and redistribute
>> it in any way they like. This is how it is, and how it should
>> be.
> 
> They can technically, yes, but the fact that they can doesn't
> map to any kind of permission for doing so.
> 
> Otherwise, anyone could take any content on the web and copy it
> freely and use it for their own purposes, without having to ask
> for permission or being restricted in the way the content is
> used.
> 
> The package authors who signed up before the change to the PyPI
> registration terms on 2009-11-29 have not given this permission:
> 
> """
> By signing up to this system, you agree to:
>  * Not upload inappropriate material (only Python packages are
>    allowed).
>  * You agree that all information you post is published (email
>    addresses are obfuscated).
> """
> 
> so being ignorant about this doesn't really help.
> 
> The authors who have signed up since were forced to accept
> terms which do give permission to anyone to do anything
> they like with the content - and that's what started this
> thread: the terms are too permissive to be acceptable.
> 
> """
> By registering to upload content to PyPI, I agree and affirmatively acknowledge the following:
> 
>    1. Content is restricted to Python packages and related information only.
>    2. Any content uploaded to PyPI is provided on a non-confidential basis.
>    3. The PSF is free to use or disseminate any content that I upload on an unrestricted basis for
> any purpose. In particular, the PSF and all other users of the web site are granted an irrevocable,
> worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform,
> and publish the content, including in digital form.
>    4. I represent and warrant that I have complied with all government regulations concerning the
> transfer or export of any content I upload to PyPI. In particular, if I am subject to United States
> law, I represent and warrant that I have obtained the proper governmental authorization for the
> export of the content I upload. I further affirm that any content I provide is not intended for use
> by a government end-user as defined in part 772 of the United States Export Administration Regulations.
> """
> 
> The version I proposed restricts those permissions to redistribution by the
> PSF - which is enough to run PyPI services:
> 
> """
> PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to
> PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following:
> 
> 1. Content is restricted to Python packages and related information only.
> 
> 2. Any content uploaded to PyPI is provided on a non-confidential basis.
> 
> 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce,
> distribute, transmit, display, perform, and publish the content, including in digital form. This
> licence is for the sole purpose of enabling the PSF to display, distribute and promote the content
> on PyPI.
> 
> 4. I represent and warrant that I have complied with all government regulations concerning the
> transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if
> I am subject to United States law, I represent and warrant that I have obtained the proper
> governmental authorization for the export of the content I upload. I further affirm that any content
> I provide is not intended for use by a government end-user as defined in part 772 of the United
> States Export Administration Regulations.
> """
> 

In (4) I would change "... upload. I further affirm" to "... upload, and
further affirm". Incorporating both assertions into a single sentence
clarifies that only those subject to US law are required to make that
declaration. The exception seems prudent given that the PSF is
incorporated in the USA.

The second version does seem much more user-friendly, somehow, and
should calm fears about potential abuse of content by the Foundation.

Are we going to go with that?

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010  http://us.pycon.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/

From martin at v.loewis.de  Sat Jan 23 14:40:21 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 23 Jan 2010 14:40:21 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5AF61E.10004@holdenweb.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>
	<4B5AC40D.4030808@egenix.com> <4B5AC97E.9010309@v.loewis.de>
	<4B5AF055.3080505@egenix.com> <4B5AF61E.10004@holdenweb.com>
Message-ID: <4B5AFC45.4070206@v.loewis.de>

> The second version does seem much more user-friendly, somehow, and
> should calm fears about potential abuse of content by the Foundation.
> 
> Are we going to go with that?

Not without legal advise. I would hope that users will also get
permission to make copies of the content uploaded to PyPI, without
having to study the license conditions of each and every packet
that they may want to copy (and they may actually have to make a copy
first to find out what the licensing conditions are).

So ISTM that MAL's version is *less* user-friendly (at least, to the
users of PyPI who actually want to use the packages that get uploaded).

Regards,
Martin

From mal at egenix.com  Sat Jan 23 14:46:51 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 23 Jan 2010 14:46:51 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5AF61E.10004@holdenweb.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>	<4B5AC40D.4030808@egenix.com>
	<4B5AC97E.9010309@v.loewis.de>	<4B5AF055.3080505@egenix.com>
	<4B5AF61E.10004@holdenweb.com>
Message-ID: <4B5AFDCB.40500@egenix.com>

Steve Holden wrote:
> M.-A. Lemburg wrote:
>> The version I proposed restricts those permissions to redistribution by the
>> PSF - which is enough to run PyPI services:
>>
>> """
>> PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to
>> PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following:
>>
>> 1. Content is restricted to Python packages and related information only.
>>
>> 2. Any content uploaded to PyPI is provided on a non-confidential basis.
>>
>> 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce,
>> distribute, transmit, display, perform, and publish the content, including in digital form. This
>> licence is for the sole purpose of enabling the PSF to display, distribute and promote the content
>> on PyPI.
>>
>> 4. I represent and warrant that I have complied with all government regulations concerning the
>> transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if
>> I am subject to United States law, I represent and warrant that I have obtained the proper
>> governmental authorization for the export of the content I upload. I further affirm that any content
>> I provide is not intended for use by a government end-user as defined in part 772 of the United
>> States Export Administration Regulations.
>> """
>>
> 
> In (4) I would change "... upload. I further affirm" to "... upload, and
> further affirm". Incorporating both assertions into a single sentence
> clarifies that only those subject to US law are required to make that
> declaration. The exception seems prudent given that the PSF is
> incorporated in the USA.

That's a tricky one: That extra sentence "I further affirm..."
introduces a restriction that goes beyond what US developers
normally have to follow. And the way it is written, it also
applies to developers not affected by US law.

However, that restriction basically says that PyPI package
may never be intended for use by government end-users, which
IMHO goes way too far - we have quite a few government users...

I'd just drop that extra limitation, since the first sentence
already covers all restrictions that a government may have
imposed on such uploads.

> The second version does seem much more user-friendly, somehow, and
> should calm fears about potential abuse of content by the Foundation.
> 
> Are we going to go with that?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 23 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Sat Jan 23 15:33:00 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 23 Jan 2010 15:33:00 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5AFC45.4070206@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>	<4B5AC40D.4030808@egenix.com>
	<4B5AC97E.9010309@v.loewis.de>	<4B5AF055.3080505@egenix.com>
	<4B5AF61E.10004@holdenweb.com> <4B5AFC45.4070206@v.loewis.de>
Message-ID: <4B5B089C.9080903@egenix.com>

"Martin v. L?wis" wrote:
>> The second version does seem much more user-friendly, somehow, and
>> should calm fears about potential abuse of content by the Foundation.
>>
>> Are we going to go with that?
> 
> Not without legal advise. I would hope that users will also get
> permission to make copies of the content uploaded to PyPI, without
> having to study the license conditions of each and every packet
> that they may want to copy (and they may actually have to make a copy
> first to find out what the licensing conditions are).
>
> So ISTM that MAL's version is *less* user-friendly (at least, to the
> users of PyPI who actually want to use the packages that get uploaded).

If you read the text, you'll notice that nothing will change for
PyPI users.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 23 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From tjreedy at udel.edu  Sun Jan 24 00:51:52 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 23 Jan 2010 18:51:52 -0500
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B5AC97E.9010309@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>
	<4B5AC40D.4030808@egenix.com> <4B5AC97E.9010309@v.loewis.de>
Message-ID: <hjg22k$tj3$1@ger.gmane.org>

On 1/23/2010 5:03 AM, "Martin v. L?wis" wrote:

> Indeed, that's what we are discussing here. And, I can only
> repeat myself: anybody can download the data and redistribute
> it in any way they like. This is how it is, and how it should
> be.

Many sites have the restriction that people can download for own use but 
*not* redistribute, at least not publicly. And people should *not* be 
able to 'redistribute in any way they like', for instance with claim of 
authorship or copyright or malware attached.



From martin at v.loewis.de  Sun Jan 24 00:57:35 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 24 Jan 2010 00:57:35 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI
	usage	agreement
In-Reply-To: <hjg22k$tj3$1@ger.gmane.org>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>	<4B5AC40D.4030808@egenix.com>
	<4B5AC97E.9010309@v.loewis.de> <hjg22k$tj3$1@ger.gmane.org>
Message-ID: <4B5B8CEF.80807@v.loewis.de>

Terry Reedy wrote:
> On 1/23/2010 5:03 AM, "Martin v. L?wis" wrote:
> 
>> Indeed, that's what we are discussing here. And, I can only
>> repeat myself: anybody can download the data and redistribute
>> it in any way they like. This is how it is, and how it should
>> be.
> 
> Many sites have the restriction that people can download for own use but
> *not* redistribute, at least not publicly.

Can you give examples of such sites (preferably examples where the
content is not originally owned by the site distributing it)?

> And people should *not* be
> able to 'redistribute in any way they like', for instance with claim of
> authorship or copyright or malware attached.

Either case would involve modification. Copying and creating derivative
works are fairly different, so when one is granted people wouldn't
assume to have the other as well.

Regards,
Martin

From martin at v.loewis.de  Sun Jan 24 11:20:16 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 24 Jan 2010 11:20:16 +0100
Subject: [Catalog-sig] Changes to the simple pages
In-Reply-To: <20100120005756.8D9B13A4065@sparrow.telecommunity.com>
References: <4B54CB54.9020001@v.loewis.de>
	<b654cd2e1001191208w783b84aduab03ac2bdd410c17@mail.gmail.com>
	<20100119210547.08AEE3A4065@sparrow.telecommunity.com>
	<4B562255.2050709@v.loewis.de>
	<20100120005756.8D9B13A4065@sparrow.telecommunity.com>
Message-ID: <4B5C1EE0.7060908@v.loewis.de>

> No, what I mean is that if PyPI returns *identical* HTML for requests
> going to "/foo" and "/foo/", then one of those pages will contain
> incorrect relative URLs.

Thanks a lot. I probably wouldn't have considered that; now I followed
your suggestion of adding redirects.

I have also changed the content to be well-formed XML.

If somebody encounters a problem, please let me know.

Regards,
Martin

From mal at egenix.com  Mon Jan 25 11:59:32 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 25 Jan 2010 11:59:32 +0100
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to
	PyPI	usage	agreement
In-Reply-To: <4B5B8CEF.80807@v.loewis.de>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>	<4B5AC40D.4030808@egenix.com>	<4B5AC97E.9010309@v.loewis.de>
	<hjg22k$tj3$1@ger.gmane.org> <4B5B8CEF.80807@v.loewis.de>
Message-ID: <4B5D7994.70501@egenix.com>

"Martin v. L?wis" wrote:
> Terry Reedy wrote:
>> On 1/23/2010 5:03 AM, "Martin v. L?wis" wrote:
>>
>>> Indeed, that's what we are discussing here. And, I can only
>>> repeat myself: anybody can download the data and redistribute
>>> it in any way they like. This is how it is, and how it should
>>> be.
>>
>> Many sites have the restriction that people can download for own use but
>> *not* redistribute, at least not publicly.
> 
> Can you give examples of such sites (preferably examples where the
> content is not originally owned by the site distributing it)?

Google Code - the example I used as basis for the updated terms -
is a popular OSS site.

Most commercial sites don't allow redistribution of the applications
you download - at least not without explicit permission.

Many freeware applications also restrict the way you may distribute
the apps, e.g. they often explicitly disallow putting them on CDs
in order to prevent magazines from cashing in on the drive-by-revenue.

For software containing crypto code, managing the distribution channels
is required by government law in most countries (see
http://rechten.uvt.nl/koops/cryptolaw/cls-sum.htm).

Software using GPL-like terms also put special requirements on
distribution channels, in fact, a major part of the GPL mechanism
is built around public distribution of software (which is one of
the issues some people have with it and the reason why the AGPL
was created).

The current terms on PyPI have the potential of overriding all such
restrictions.

>> And people should *not* be
>> able to 'redistribute in any way they like', for instance with claim of
>> authorship or copyright or malware attached.
> 
> Either case would involve modification. Copying and creating derivative
> works are fairly different, so when one is granted people wouldn't
> assume to have the other as well.

I think we now all know that you're not in favor of changing
anything reagrding the new terms on PyPI. That's fine.

Please do acknowledge, though, that there are a number of people
who do not feel comfortable with those new terms.

Whether or not to change them, is up to the PSF board. I've posted
a proposal which tries to find a balance between what the PSF
has to ask from the uploading developer and what the PyPI user
can reasonably expect.

It's now up to the board to decide.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 25 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From van.lindberg at gmail.com  Mon Jan 25 15:39:59 2010
From: van.lindberg at gmail.com (VanL)
Date: Mon, 25 Jan 2010 08:39:59 -0600
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5AFDCB.40500@egenix.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>	<4B5AC40D.4030808@egenix.com>
	<4B5AC97E.9010309@v.loewis.de>	<4B5AF055.3080505@egenix.com>
	<4B5AF61E.10004@holdenweb.com> <4B5AFDCB.40500@egenix.com>
Message-ID: <4B5DAD3F.6030905@gmail.com>

On 1/23/2010 7:46 AM, M.-A. Lemburg wrote:
>
> That's a tricky one: That extra sentence "I further affirm..."
> introduces a restriction that goes beyond what US developers
> normally have to follow. And the way it is written, it also
> applies to developers not affected by US law.
>
> However, that restriction basically says that PyPI package
> may never be intended for use by government end-users, which
> IMHO goes way too far - we have quite a few government users...
>
> I'd just drop that extra limitation, since the first sentence
> already covers all restrictions that a government may have
> imposed on such uploads.
>   
There are specific restrictions that mandate the additional language
about foreign government end-users. Sorry :) 

Also see MvL for the thought that went into the current wording. As I
have stated before, the wording doesn't grant the PSF the authority to
relicense or make derivative works of the content. Rather, it allows the
PSF (and mirrors, and people who use PyPI) to reproduce exactly the
content that was uploaded to PyPI.

Thanks,

Van

From mal at egenix.com  Mon Jan 25 16:57:35 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 25 Jan 2010 16:57:35 +0100
Subject: [Catalog-sig]
	[PSF-Board]	Troubled	by	changes	to	PyPI	usage	agreement
In-Reply-To: <4B5DAD3F.6030905@gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>	<4B5AC40D.4030808@egenix.com>	<4B5AC97E.9010309@v.loewis.de>	<4B5AF055.3080505@egenix.com>	<4B5AF61E.10004@holdenweb.com>
	<4B5AFDCB.40500@egenix.com> <4B5DAD3F.6030905@gmail.com>
Message-ID: <4B5DBF6F.4070301@egenix.com>

VanL wrote:
> On 1/23/2010 7:46 AM, M.-A. Lemburg wrote:
>>
>> That's a tricky one: That extra sentence "I further affirm..."
>> introduces a restriction that goes beyond what US developers
>> normally have to follow. And the way it is written, it also
>> applies to developers not affected by US law.
>>
>> However, that restriction basically says that PyPI package
>> may never be intended for use by government end-users, which
>> IMHO goes way too far - we have quite a few government users...
>>
>> I'd just drop that extra limitation, since the first sentence
>> already covers all restrictions that a government may have
>> imposed on such uploads.
>>   
> There are specific restrictions that mandate the additional language
> about foreign government end-users. Sorry :) 

Here's the complete definition of "government end-user" as defined
in part 772 of the United States Export Administration Regulations
(taken from http://www.access.gpo.gov/bis/ear/txt/772.txt):

"""
"Government end-user" (as applied to encryption
items). A government end-user is any foreign
central, regional or local government department,
agency, or other entity performing governmental
functions; including governmental research
institutions,  governmental corporations or their
separate business units (as defined in part 772 of
the EAR) which are engaged in the manufacture
or distribution of items or services controlled on
the Wassenaar Munitions List, and international
governmental organizations.  This term does not
include: utilities (including telecommunications
companies and Internet service providers); banks
and financial institutions; transportation;
broadcast or entertainment; educational
organizations; civil health and medical
organizations; retail or wholesale firms; and
manufacturing or industrial entities not engaged in
the manufacture or distribution of items or
services controlled on the Wassenaar Munitions
List.
"""

AFAICT, the above term is only used for crypto items, not any
code in general.

However, the current form of the PyPI terms clause 4 applies
to any code that could be used by e.g. parts of the military
of some foreign country (with foreign meaning non-US, I suppose).

PyPI packages will usually not have any extra use restrictions
for the above "government end-users", so requesting from a package
author to "affirm that any content I provide is not intended for
use by a government end-user" basically requires that:

a) the author adds a special restriction to the content (even if
   the code doesn't contain any crypto items), or

b) doesn't upload the content.

OTOH, by removing that extra sentence, only the first sentence of clause
4 applies, which does allow such uploads for non-crypto code and
only mandates the restrictions related to foreign government
end-users as defined by the US EAR (with all its complex rules).

If we then add new PyPI user terms which disallow use of PyPI
in ways that are prohibited by the US EAR and whatever is
mandated by The Netherlands, we'd create a much cleaner situation
for everyone.

> Also see MvL for the thought that went into the current wording. As I
> have stated before, the wording doesn't grant the PSF the authority to
> relicense or make derivative works of the content. Rather, it allows the
> PSF (and mirrors, and people who use PyPI) to reproduce exactly the
> content that was uploaded to PyPI.

Giving the PSF those redistribution right is not the problem.

It's giving all web site users those same redistribution rights
that's causing trouble.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 25 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From tjreedy at udel.edu  Mon Jan 25 21:35:10 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Jan 2010 15:35:10 -0500
Subject: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage
	agreement
In-Reply-To: <4B5DAD3F.6030905@gmail.com>
References: <87y6ljhy8h.fsf@benfinney.id.au>	<C1D58890-97EF-4FDE-B526-E35583FABE84@gmail.com>	<4B193350.7090609@python.org>	<4B1961D1.7010405@egenix.com>	<019d01ca7518$2dba00f0$892e02d0$@net>	<4B1D5CCB.4000309@egenix.com>	<4B1D61C3.7030705@python.org>	<4B1D7601.5040608@egenix.com>	<4B1DB9BE.5040000@python.org>	<4B1FE7B1.9030608@egenix.com>	<4B54D896.2060108@egenix.com>	<4B553366.2090809@holdenweb.com>	<4B56D0C0.2080703@egenix.com>	<4B572088.7020501@holdenweb.com>	<4B5778D8.9090402@v.loewis.de>	<4B577BA3.7070606@egenix.com>	<4B577CE9.2050809@v.loewis.de>	<4B57807F.2050303@egenix.com>	<4B5786ED.2020708@v.loewis.de>	<4B583398.9090509@egenix.com>	<4B58BF9C.6060403@v.loewis.de>	<4B59744A.7040103@egenix.com>	<4B5A227D.4040307@v.loewis.de>	<4B5AC40D.4030808@egenix.com>	<4B5AC97E.9010309@v.loewis.de>	<4B5AF055.3080505@egenix.com>	<4B5AF61E.10004@holdenweb.com>
	<4B5AFDCB.40500@egenix.com> <4B5DAD3F.6030905@gmail.com>
Message-ID: <hjkv9u$s07$1@ger.gmane.org>

On 1/25/2010 9:39 AM, VanL wrote:

> Also see MvL for the thought that went into the current wording. As I
> have stated before, the wording doesn't grant the PSF the authority to
> relicense or make derivative works of the content. Rather, it allows the
> PSF (and mirrors, and people who use PyPI) to reproduce exactly the
> content that was uploaded to PyPI.

If that is what you mean, then why are not you willing to say exactly 
that, instead of 'unrestricted ...', which to me means 'unrestricted'.

I checked the Flickr Terms of Service and it pretty much says just what 
you say above. To paraphrase, "By uploading content, you grant us all 
the rights we need to distribute it via any of our servers as part of 
the normal operation of our web site service."

Flickr is also more fair about the term of the agreement. Since they do 
not want to incur a perpetual obligation and indeed want the right to 
cease distribution and even terminate accounts at will, they allow Users 
to do the same.

Terry Jan Reedy