From martin at v.loewis.de  Sun Sep 12 11:57:18 2010
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 12 Sep 2010 11:57:18 +0200
Subject: [Catalog-sig] RSS 2.0
Message-ID: <4C8CA3FE.7070005@v.loewis.de>

In

https://sourceforge.net/tracker/index.php?func=detail&aid=2987704&group_id=66150&atid=513506

somebody requested that the PyPI RSS should add the enclosure element to
item, pointing to the download URL. IIUC, this would require changing
the RSS to 2.0.

So: Should I make this change?

Regards,
Martin

From ben+python at benfinney.id.au  Sun Sep 12 15:05:10 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 12 Sep 2010 23:05:10 +1000
Subject: [Catalog-sig] RSS 2.0
References: <4C8CA3FE.7070005@v.loewis.de>
Message-ID: <87aann8649.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> somebody requested that the PyPI RSS should add the enclosure element
> to item, pointing to the download URL. IIUC, this would require
> changing the RSS to 2.0.
>
> So: Should I make this change?

If changing the protocol used is an option, would it not be better to
use an actual specified standard: the Atom syndication protocol
<URL:http://en.wikipedia.org/wiki/Atom_%28standard%29>?

-- 
 \       ?We must respect the other fellow's religion, but only in the |
  `\       sense and to the extent that we respect his theory that his |
_o__)     wife is beautiful and his children smart.? ?Henry L. Mencken |
Ben Finney


From martin at v.loewis.de  Sun Sep 12 18:57:53 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 12 Sep 2010 18:57:53 +0200
Subject: [Catalog-sig] RSS 2.0
In-Reply-To: <87aann8649.fsf@benfinney.id.au>
References: <4C8CA3FE.7070005@v.loewis.de> <87aann8649.fsf@benfinney.id.au>
Message-ID: <4C8D0691.7020805@v.loewis.de>

>> So: Should I make this change?
> 
> If changing the protocol used is an option, would it not be better to
> use an actual specified standard: the Atom syndication protocol
> <URL:http://en.wikipedia.org/wiki/Atom_%28standard%29>?

For the issue at hand, certainly not: Atom does not have the enclosure
element.

Regards,
Martin

From jcea at jcea.es  Mon Sep 13 04:30:48 2010
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 13 Sep 2010 04:30:48 +0200
Subject: [Catalog-sig] RSS 2.0
In-Reply-To: <87aann8649.fsf@benfinney.id.au>
References: <4C8CA3FE.7070005@v.loewis.de> <87aann8649.fsf@benfinney.id.au>
Message-ID: <4C8D8CD8.9090706@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/09/10 15:05, Ben Finney wrote:
> If changing the protocol used is an option, would it not be better to
> use an actual specified standard: the Atom syndication protocol
> <URL:http://en.wikipedia.org/wiki/Atom_%28standard%29>?

+1 to ATOM.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTI2M2Jlgi5GaxT1NAQK9qgP9HnMupqU/XULhxNDDNvpOp9cpT5uJfPpl
lOytnirdRwum03dZiV0IY4StHTpGKaqo6rfw/vxRcyoaMX6rEB1nCH8pnknhlkpU
ozXS2gFgwB1E+ve8Q/MEz9L1ehzpEXqtl8HmPHlplBht7Du6roJEu+Dvv0cyufke
CV/RPvnQ3zw=
=Nmwl
-----END PGP SIGNATURE-----

From ben+python at benfinney.id.au  Mon Sep 13 06:01:36 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 13 Sep 2010 14:01:36 +1000
Subject: [Catalog-sig] PyPI syndication format (was: RSS 2.0)
References: <4C8CA3FE.7070005@v.loewis.de>
Message-ID: <87y6b670m7.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> https://sourceforge.net/tracker/index.php?func=detail&aid=2987704&group_id=66150&atid=513506

I can't read that to see what the details are. The page returned says:

    ERROR
    Artifact: This Artifact Has Been Made Private. Only Group Members Can
    View Private ArtifactTypes. 

"Martin v. L?wis" <martin at v.loewis.de> writes:

> > If changing the protocol used is an option, would it not be better
> > to use an actual specified standard: the Atom syndication protocol
> > <URL:http://en.wikipedia.org/wiki/Atom_%28standard%29>?
>
> For the issue at hand, certainly not: Atom does not have the enclosure
> element.

Which syndication are we talking about? I don't see enclosures used for
<URL:http://pypi.python.org/pypi?:action=rss>, which seems quite
replaceable with standard Atom syndication.

If the issue you're referring to is described at the above Sourceforge
URL, then someone with access to it will need to summarise what the
salient points are.

Regardless, it seems that the main syndication could switch to standard
Atom syndication without losing useful information.

-- 
 \      ?How many people here have telekenetic powers? Raise my hand.? |
  `\                                                      ?Emo Philips |
_o__)                                                                  |
Ben Finney


From martin at v.loewis.de  Mon Sep 13 08:17:34 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 13 Sep 2010 08:17:34 +0200
Subject: [Catalog-sig] PyPI syndication format
In-Reply-To: <87y6b670m7.fsf@benfinney.id.au>
References: <4C8CA3FE.7070005@v.loewis.de> <87y6b670m7.fsf@benfinney.id.au>
Message-ID: <4C8DC1FE.7060403@v.loewis.de>

Am 13.09.2010 06:01, schrieb Ben Finney:
> "Martin v. L?wis" <martin at v.loewis.de> writes:
> 
>> https://sourceforge.net/tracker/index.php?func=detail&aid=2987704&group_id=66150&atid=513506
> 
> I can't read that to see what the details are. The page returned says:
> 
>     ERROR
>     Artifact: This Artifact Has Been Made Private. Only Group Members Can
>     View Private ArtifactTypes. 

Oops. Please try

https://sourceforge.net/tracker/index.php?func=detail&aid=2987704&group_id=66150&atid=513503

>> For the issue at hand, certainly not: Atom does not have the enclosure
>> element.
> 
> Which syndication are we talking about? I don't see enclosures used for
> <URL:http://pypi.python.org/pypi?:action=rss>

See my original posting. The request is to add them.

> Regardless, it seems that the main syndication could switch to standard
> Atom syndication without losing useful information.

But certainly not on the same URL, right? Speaking in favor of Atom
doesn't help me to resolve the issue at hand.

Regards,
Martin

From ben+python at benfinney.id.au  Mon Sep 13 08:37:31 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 13 Sep 2010 16:37:31 +1000
Subject: [Catalog-sig] PyPI syndication format
References: <4C8CA3FE.7070005@v.loewis.de> <87y6b670m7.fsf@benfinney.id.au>
	<4C8DC1FE.7060403@v.loewis.de>
Message-ID: <87tylu6tec.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> Am 13.09.2010 06:01, schrieb Ben Finney:

> > Which syndication are we talking about? I don't see enclosures used
> > for <URL:http://pypi.python.org/pypi?:action=rss>
>
> See my original posting. The request is to add them.

Ah okay. Now that you've shown the issue URL that we can read :-) I can
see that request.

> > "Martin v. L?wis" <martin at v.loewis.de> writes:
> >> For the issue at hand, certainly not: Atom does not have the
> >> enclosure element.

A ?link? element can be declared an enclosure by specifying a
?rel="enclosure"? attribute, according to RFC4287 ?4.2.7.2
<URL:http://tools.ietf.org/html/rfc4287#section-4.2.7.2>.

> But certainly not on the same URL, right? Speaking in favor of Atom
> doesn't help me to resolve the issue at hand.

I'm not sure what this means (?on the same URL?). Does the above
clarification help?

-- 
 \     ?What is needed is not the will to believe but the will to find |
  `\              out, which is the exact opposite.? ?Bertrand Russell |
_o__)                                                                  |
Ben Finney


From martin at v.loewis.de  Mon Sep 13 08:55:11 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 13 Sep 2010 08:55:11 +0200
Subject: [Catalog-sig] PyPI syndication format
In-Reply-To: <87tylu6tec.fsf@benfinney.id.au>
References: <4C8CA3FE.7070005@v.loewis.de>
	<87y6b670m7.fsf@benfinney.id.au>	<4C8DC1FE.7060403@v.loewis.de>
	<87tylu6tec.fsf@benfinney.id.au>
Message-ID: <4C8DCACF.5010801@v.loewis.de>

>>> "Martin v. L?wis" <martin at v.loewis.de> writes:
>>>> For the issue at hand, certainly not: Atom does not have the
>>>> enclosure element.
> 
> A ?link? element can be declared an enclosure by specifying a
> ?rel="enclosure"? attribute, according to RFC4287 ?4.2.7.2
> <URL:http://tools.ietf.org/html/rfc4287#section-4.2.7.2>.
> 
>> But certainly not on the same URL, right? Speaking in favor of Atom
>> doesn't help me to resolve the issue at hand.
> 
> I'm not sure what this means (?on the same URL?). Does the above
> clarification help?

No. Currently, the RSS is provided at

http://pypi.python.org/pypi?%3Aaction=rss

Are you proposing that, when accessing this very URL, an Atom
content should be returned?

If not: Are you in favor, opposed to, or indifferent to the
proposal of adding the enclosure element to the RSS, along
with incrementing the RSS version number?

Regards,
Martin

From ben+python at benfinney.id.au  Mon Sep 13 09:12:18 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 13 Sep 2010 17:12:18 +1000
Subject: [Catalog-sig] PyPI syndication format
References: <4C8CA3FE.7070005@v.loewis.de> <87y6b670m7.fsf@benfinney.id.au>
	<4C8DC1FE.7060403@v.loewis.de> <87tylu6tec.fsf@benfinney.id.au>
	<4C8DCACF.5010801@v.loewis.de>
Message-ID: <87pqwi6rsd.fsf@benfinney.id.au>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> > I'm not sure what this means (?on the same URL?). Does the above
> > clarification help?
>
> No. Currently, the RSS is provided at
> http://pypi.python.org/pypi?%3Aaction=rss
>
> Are you proposing that, when accessing this very URL, an Atom
> content should be returned?

Ah, I understand. No, it would be inappropriate for that URL to serve an
Atom document.

> If not: Are you in favor, opposed to, or indifferent to the proposal
> of adding the enclosure element to the RSS, along with incrementing
> the RSS version number?

I'm in favour of the issue reported originally be solved by skipping any
changes to the RSS altogether (leave the existing one for a time, of
course) and using Atom syndication, since AFAICT it provides all the
requirements discussed so far but in addition is properly specified and
standardised.

-- 
 \     ?The cost of a thing is the amount of what I call life which is |
  `\       required to be exchanged for it, immediately or in the long |
_o__)                                       run.? ?Henry David Thoreau |
Ben Finney


From martin at v.loewis.de  Mon Sep 13 09:40:32 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 13 Sep 2010 09:40:32 +0200
Subject: [Catalog-sig] PyPI syndication format
In-Reply-To: <87pqwi6rsd.fsf@benfinney.id.au>
References: <4C8CA3FE.7070005@v.loewis.de>
	<87y6b670m7.fsf@benfinney.id.au>	<4C8DC1FE.7060403@v.loewis.de>
	<87tylu6tec.fsf@benfinney.id.au>	<4C8DCACF.5010801@v.loewis.de>
	<87pqwi6rsd.fsf@benfinney.id.au>
Message-ID: <4C8DD570.4090504@v.loewis.de>

> I'm in favour of the issue reported originally be solved by skipping any
> changes to the RSS altogether (leave the existing one for a time, of
> course) and using Atom syndication, since AFAICT it provides all the
> requirements discussed so far but in addition is properly specified and
> standardised.

I don't think this ("all the requirements discussed so far") is actually
the case. In the report, the reporter specifically asks for AppCast
support, and elaborates that this is what

http://connectedflow.com/appcasting/

says, which in turn says

# Appcasting is the practice of using the 'enclosure' feature of RSS
# 2.0 feeds

So, by nature, Atom can't possibly support Appcasting.

I also wonder why you keep saying that RSS is not properly specified.

http://cyber.law.harvard.edu/rss/rss.html

looks like a fine specification to me (but this really is a side issue;
another is the use of the word "syndication": I don't know what it
means, I then always think of organized crime and corruption - PyPI
is not a member of any syndicate).

Regards,
Martin

From fuzzyman at voidspace.org.uk  Mon Sep 13 11:59:01 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 13 Sep 2010 10:59:01 +0100
Subject: [Catalog-sig] RSS 2.0
In-Reply-To: <4C8D0691.7020805@v.loewis.de>
References: <4C8CA3FE.7070005@v.loewis.de> <87aann8649.fsf@benfinney.id.au>
	<4C8D0691.7020805@v.loewis.de>
Message-ID: <AANLkTi=7xYRO=DWO=C+-xTiOLN1i=F439Sx-nbTCO945@mail.gmail.com>

On 12 September 2010 17:57, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> >> So: Should I make this change?
> >
> > If changing the protocol used is an option, would it not be better to
> > use an actual specified standard: the Atom syndication protocol
> > <URL:http://en.wikipedia.org/wiki/Atom_%28standard%29>?
>
> For the issue at hand, certainly not: Atom does not have the enclosure
> element.
>
>
According to this article Atom supports enclosures through <link
rel="enclosure".... />

http://www.ibm.com/developerworks/xml/library/x-atom10.html#N100EB

Michael



> Regards,
> Martin
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>



-- 
http://www.voidspace.org.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100913/1ca9eb52/attachment.html>

From tseaver at palladion.com  Mon Sep 13 20:16:01 2010
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 13 Sep 2010 14:16:01 -0400
Subject: [Catalog-sig] PyPI syndication format
In-Reply-To: <4C8DD570.4090504@v.loewis.de>
References: <4C8CA3FE.7070005@v.loewis.de>	<87y6b670m7.fsf@benfinney.id.au>	<4C8DC1FE.7060403@v.loewis.de>	<87tylu6tec.fsf@benfinney.id.au>	<4C8DCACF.5010801@v.loewis.de>	<87pqwi6rsd.fsf@benfinney.id.au>
	<4C8DD570.4090504@v.loewis.de>
Message-ID: <i6lpou$ga7$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin v. L?wis wrote:
>> I'm in favour of the issue reported originally be solved by skipping any
>> changes to the RSS altogether (leave the existing one for a time, of
>> course) and using Atom syndication, since AFAICT it provides all the
>> requirements discussed so far but in addition is properly specified and
>> standardised.
> 
> I don't think this ("all the requirements discussed so far") is actually
> the case. In the report, the reporter specifically asks for AppCast
> support, and elaborates that this is what
> 
> http://connectedflow.com/appcasting/
> 
> says, which in turn says
> 
> # Appcasting is the practice of using the 'enclosure' feature of RSS
> # 2.0 feeds
> 
> So, by nature, Atom can't possibly support Appcasting.
> 
> I also wonder why you keep saying that RSS is not properly specified.
> 
> http://cyber.law.harvard.edu/rss/rss.html

Atom addresses a number of ambiguities in the RSS 2.0 spec.  See:

 http://www.intertwingly.net/wiki/pie/Rss20AndAtom10Compared

In particular, handling markup (particularly HTML) as the content of
tags is underspecified in RSS 2.0.

> looks like a fine specification to me (but this really is a side issue;
> another is the use of the word "syndication": I don't know what it
> means, I then always think of organized crime and corruption - PyPI
> is not a member of any syndicate).

"Content syndication" is a fairly common term:  it has pre-internet uses
for stuff like "syndicated columns" in newspapers.  See:

 http://en.wikipedia.org/wiki/Web_syndication


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkyOalwACgkQ+gerLs4ltQ5ElwCbBrOZHSEH/5IJBfU0lyq6UJlP
fKwAoMhTeIqI24bzRjQbS6FE0s3BWMdR
=FFiI
-----END PGP SIGNATURE-----


From martin at v.loewis.de  Mon Sep 13 22:15:11 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 13 Sep 2010 22:15:11 +0200
Subject: [Catalog-sig] RSS 2.0
In-Reply-To: <AANLkTi=7xYRO=DWO=C+-xTiOLN1i=F439Sx-nbTCO945@mail.gmail.com>
References: <4C8CA3FE.7070005@v.loewis.de>	<87aann8649.fsf@benfinney.id.au>	<4C8D0691.7020805@v.loewis.de>
	<AANLkTi=7xYRO=DWO=C+-xTiOLN1i=F439Sx-nbTCO945@mail.gmail.com>
Message-ID: <4C8E864F.1040104@v.loewis.de>

Am 13.09.2010 11:59, schrieb Michael Foord:
> 
> 
> On 12 September 2010 17:57, "Martin v. L?wis" <martin at v.loewis.de
> <mailto:martin at v.loewis.de>> wrote:
> 
>     >> So: Should I make this change?
>     >
>     > If changing the protocol used is an option, would it not be better to
>     > use an actual specified standard: the Atom syndication protocol
>     > <URL:http://en.wikipedia.org/wiki/Atom_%28standard%29>?
> 
>     For the issue at hand, certainly not: Atom does not have the enclosure
>     element.
> 
> 
> According to this article Atom supports enclosures through <link
> rel="enclosure".... />
> 
> http://www.ibm.com/developerworks/xml/library/x-atom10.html#N100EB

Sure. However, I fail to see how this helps in resolving the issue
at hand: Should PyPI support AppCast or not?

Regards,
Martin


From doug.hellmann at gmail.com  Mon Sep 13 22:36:29 2010
From: doug.hellmann at gmail.com (Doug Hellmann)
Date: Mon, 13 Sep 2010 16:36:29 -0400
Subject: [Catalog-sig] RSS 2.0
In-Reply-To: <4C8E864F.1040104@v.loewis.de>
References: <4C8CA3FE.7070005@v.loewis.de>	<87aann8649.fsf@benfinney.id.au>	<4C8D0691.7020805@v.loewis.de>
	<AANLkTi=7xYRO=DWO=C+-xTiOLN1i=F439Sx-nbTCO945@mail.gmail.com>
	<4C8E864F.1040104@v.loewis.de>
Message-ID: <946193BB-D1AA-4A60-97F2-74D515ACF734@gmail.com>


On Sep 13, 2010, at 4:15 PM, Martin v. L?wis wrote:

> Am 13.09.2010 11:59, schrieb Michael Foord:
>> 
>> 
>> On 12 September 2010 17:57, "Martin v. L?wis" <martin at v.loewis.de
>> <mailto:martin at v.loewis.de>> wrote:
>> 
>>>> So: Should I make this change?
>>> 
>>> If changing the protocol used is an option, would it not be better to
>>> use an actual specified standard: the Atom syndication protocol
>>> <URL:http://en.wikipedia.org/wiki/Atom_%28standard%29>?
>> 
>>    For the issue at hand, certainly not: Atom does not have the enclosure
>>    element.
>> 
>> 
>> According to this article Atom supports enclosures through <link
>> rel="enclosure".... />
>> 
>> http://www.ibm.com/developerworks/xml/library/x-atom10.html#N100EB
> 
> Sure. However, I fail to see how this helps in resolving the issue
> at hand: Should PyPI support AppCast or not?

Yes, I think it would be useful.

Doug


From martin at v.loewis.de  Fri Sep 17 20:43:36 2010
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 17 Sep 2010 20:43:36 +0200
Subject: [Catalog-sig] egg_info in PyPI
Message-ID: <4C93B6D8.1090700@v.loewis.de>

Here at the DZUG conference, we are planning to integrate explicit 
access to setuptools metadata to the package index.

The current idea is to store the contents of the egg_info directory,
giving remote access to specific files. By default, PyPI will extract,
per release, data from the egg that may get uploaded (using the first
one if multiple eggs get uploaded). If no egg gets uploaded, a VM
based build service will generate it from a source distributions.
Tools like setuptools or distribute could also directly upload this
information, e.g. as part of the register command.

Any opinions?

Regards,
Martin

From jim at zope.com  Fri Sep 17 20:58:41 2010
From: jim at zope.com (Jim Fulton)
Date: Fri, 17 Sep 2010 14:58:41 -0400
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C93B6D8.1090700@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
Message-ID: <AANLkTikLhhy7AYt3i6nAmX+UksqanNtUfFcueFMCRgeB@mail.gmail.com>

On Fri, Sep 17, 2010 at 2:43 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Here at the DZUG conference, we are planning to integrate explicit access to
> setuptools metadata to the package index.
>
> The current idea is to store the contents of the egg_info directory,
> giving remote access to specific files. By default, PyPI will extract,
> per release, data from the egg that may get uploaded (using the first
> one if multiple eggs get uploaded). If no egg gets uploaded, a VM
> based build service will generate it from a source distributions.
> Tools like setuptools or distribute could also directly upload this
> information, e.g. as part of the register command.
>
> Any opinions?

Having that information, especially dependency information would allow
much better package assembly than we have now.

So an enthusiastic +1. :)

Jim

-- 
Jim Fulton

From ziade.tarek at gmail.com  Fri Sep 17 21:18:06 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 17 Sep 2010 21:18:06 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C93B6D8.1090700@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
Message-ID: <AANLkTi=_FGDGykUPsXWCKunpfcigJf8Y8jtix=S5MSw4@mail.gmail.com>

Since egg_info results may vary depending on the platform used to built it,
how do you plan to do this so people can use these files ?

That s what we are trying to solve in pep345 by having context free metada

Regards
Tarek

On Sep 17, 2010 8:43 PM, Martin v. L?wis <martin at v.loewis.de> wrote:

Here at the DZUG conference, we are planning to integrate explicit access to
setuptools metadata to the package index.

The current idea is to store the contents of the egg_info directory,
giving remote access to specific files. By default, PyPI will extract,
per release, data from the egg that may get uploaded (using the first
one if multiple eggs get uploaded). If no egg gets uploaded, a VM
based build service will generate it from a source distributions.
Tools like setuptools or distribute could also directly upload this
information, e.g. as part of the register command.

Any opinions?

Regards,
Martin
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG at python.org
http://mail.python.org/mailman/listinfo/catalog-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100917/e85877e5/attachment.html>

From pje at telecommunity.com  Fri Sep 17 21:27:16 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 17 Sep 2010 15:27:16 -0400
Subject: [Catalog-sig] egg_info in PyPI
Message-ID: <20100917192717.964013A403D@sparrow.telecommunity.com>

At 08:43 PM 9/17/2010 +0200, Martin v. L?wis wrote:
>Here at the DZUG conference, we are planning to integrate explicit 
>access to setuptools metadata to the package index.
>
>The current idea is to store the contents of the egg_info directory,
>giving remote access to specific files. By default, PyPI will extract,
>per release, data from the egg that may get uploaded (using the first
>one if multiple eggs get uploaded). If no egg gets uploaded, a VM
>based build service will generate it from a source distributions.
>Tools like setuptools or distribute could also directly upload this
>information, e.g. as part of the register command.
>
>Any opinions?

FYI, egg-info directories can store arbitrary data (see e.g. the 
EggTranslations package, which uses it for localization resources), 
so you may want to impose some restrictions on *which* metadata files 
to include.

Second, if a user uploads a source distribution built with 
setuptools, it will include an .egg-info directory automatically, so 
you can simply extract it from there.  Conversely, if the source 
distribution is *not* built with setuptools, then building it with 
setuptools would not produce much information in the egg-info anyway.

IOW, there might not be much point to using a build service.  A 
project that doesn't use setuptools would have (at most) these 
metadata entries:

PKG-INFO
SOURCES.txt    # a manifest of the sdist contents
top_level.txt  # list of top-level package/module names
scripts/       # source code of scripts in the package)
zip-safe or not-zip-safe   # flag file

I don't think that most of these are useful for PyPI searches, though 
I suppose a listing of the name of the scripts the package includes 
could be useful.

I'm not sure how useful it is to just have URLs for accessing the 
files, though, vs. having actual searches on structured data provided 
in the files.  For example, an index of projects by package or module 
names, or of projects that provide a particular entry point.


From jannis at leidel.info  Fri Sep 17 22:02:10 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Fri, 17 Sep 2010 22:02:10 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C93B6D8.1090700@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
Message-ID: <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>

On 17.09.2010, at 20:43, Martin v. L?wis wrote:

> Here at the DZUG conference, we are planning to integrate explicit access to setuptools metadata to the package index.
> 
> The current idea is to store the contents of the egg_info directory,
> giving remote access to specific files. By default, PyPI will extract,
> per release, data from the egg that may get uploaded (using the first
> one if multiple eggs get uploaded). If no egg gets uploaded, a VM
> based build service will generate it from a source distributions.
> Tools like setuptools or distribute could also directly upload this
> information, e.g. as part of the register command.
> 
> Any opinions?

I'm confused, wouldn't that basically be a slap in the face for the people having worked on PEP345 and distutils2, especially during the Summer of Code?

Also, and I understand enthusiasm tends to build up during conferences, but wouldn't supporting setuptools' egg-info directory again be a step backwards after all those months of discussion about the direction of Python packaging?

Jannis

From ziade.tarek at gmail.com  Sat Sep 18 01:04:18 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 18 Sep 2010 01:04:18 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
Message-ID: <AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>

On Fri, Sep 17, 2010 at 10:02 PM, Jannis Leidel <jannis at leidel.info> wrote:
> On 17.09.2010, at 20:43, Martin v. L?wis wrote:
>
>> Here at the DZUG conference, we are planning to integrate explicit access to setuptools metadata to the package index.
>>
>> The current idea is to store the contents of the egg_info directory,
>> giving remote access to specific files. By default, PyPI will extract,
>> per release, data from the egg that may get uploaded (using the first
>> one if multiple eggs get uploaded). If no egg gets uploaded, a VM
>> based build service will generate it from a source distributions.
>> Tools like setuptools or distribute could also directly upload this
>> information, e.g. as part of the register command.
>>
>> Any opinions?
>
> I'm confused, wouldn't that basically be a slap in the face for the people having worked on PEP345 and distutils2, especially during the Summer of Code?
>
> Also, and I understand enthusiasm tends to build up during conferences, but wouldn't supporting setuptools' egg-info directory again be a step backwards after all those months of discussion about the direction of Python packaging?

Yeah, we worked on a new standard that was accepted - PEP 345

PyPI is currently publishing pep 345 info as a matter of fact - I did
the patch and there's one package that already uses it
http://pypi.python.org/pypi/Distutils2.  (no deps on this one, but
other stuff like links..)

I am about to release the work we did during GSOC in distutils2, a
first beta that includes all the work we done.

Now you want to publish another metadata format at PyPI ? If PyPI
takes that direction and adopts, promotes and publishes a standard
that is not the one we worked on in the past year, this will increase
our difficulty to push the new format so its adopted by the tools then
the community.

People will just get confused because they will find two competing
metadata formats  That's exactly the situation where we were at, and
that's exactly where I don't want to go back.

I am not even understanding what's the benefit of doing this since an
egg_info directory is obtained at *build* time and can differ from a
machine to another, so it seems pretty useless for me to publish this.

The whole point of PEP 345 is to extend our metadata to statically
provide dependencies at PyPI, thanks to a micro-language that allows
you to describe dependencies for any platform.

We worked hard to build some standards, but if PyPI doesn't help us
here, everything we did and are doing is useless.

Tarek

-- 
Tarek Ziad? | http://ziade.org

From sridharr at activestate.com  Sat Sep 18 02:23:01 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Fri, 17 Sep 2010 17:23:01 -0700
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
Message-ID: <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>


On 2010-09-17, at 4:04 PM, Tarek Ziad? wrote:

> I am not even understanding what's the benefit of doing this since an
> egg_info directory is obtained at *build* time and can differ from a
> machine to another, so it seems pretty useless for me to publish this.

I am in full agreement with Tarek here. At ActiveState, we maintain our own index that differs from PyPI in two ways (among others):

- use setuptools.package_index to scrap sdists for packages that don't upload them to pypi
- PKG-INFO and requires.txt are extracted (if doesn't exist, generate using egg_info cmd) from each of the sdist

(and then our index provides the full metadata - with internal links to sdists - as a sqlite db for the builder processes on each platform)

The problem with extracting PKG-INFO and requires.txt on the index server is that, the contents of requires.txt sometimes differ based on which platform and Python version on which the egg_info command was run. For eg., the "tox" project depends[1] on "virtualenv" package if it is run using Python2, but not on Python3.

> The whole point of PEP 345 is to extend our metadata to statically
> provide dependencies at PyPI, thanks to a micro-language that allows
> you to describe dependencies for any platform.

Static metadata would allow packages like "tox" to configure Python version / platform specific dependencies without resorting to runtime checks. 

> We worked hard to build some standards, but if PyPI doesn't help us
> here, everything we did and are doing is useless.

Ideally, in future - I should be able to query static metadata (with environment markers[2] and such) for *any* package from PyPI. And this static metadata is simply a DIST-INFO file (instead of being a directory with a bunch of files in it). I don't really see a point in providing access to, say, the list of entry points of each package. As for as package managers is concerned, the only things that matter are a) list of package names and versions, b) source tarball for each release c) and the corresponding metadata with dependency information.

-srid

[1] http://code.google.com/p/pytox/source/browse/setup.py#30
[2] http://www.python.org/dev/peps/pep-0345/#environment-markers

From martin at v.loewis.de  Sat Sep 18 09:31:49 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 09:31:49 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <AANLkTi=_FGDGykUPsXWCKunpfcigJf8Y8jtix=S5MSw4@mail.gmail.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<AANLkTi=_FGDGykUPsXWCKunpfcigJf8Y8jtix=S5MSw4@mail.gmail.com>
Message-ID: <4C946AE5.8020802@v.loewis.de>

> Since egg_info results may vary depending on the platform used to built
> it, how do you plan to do this so people can use these files ?

See below for how exactly the data is provided (in particular, the 
oldest egg is used, unless the user explicitly posts other data).

Whether or not the data is useful to anybody else, I don't know
(I don't use eggs myself).
Jim said it will be useful to him, so did a number of other people.
PyPI itself will not try to interpret the data in any way.

> That s what we are trying to solve in pep345 by having context free metada

Sure. I expect that this service remains useful only until all packages
have been converted to PEP 345.

Regards,
Martin

From martin at v.loewis.de  Sat Sep 18 09:41:35 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 09:41:35 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <20100917191808.3DAF03A4100@sparrow.telecommunity.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<20100917191808.3DAF03A4100@sparrow.telecommunity.com>
Message-ID: <4C946D2F.3040202@v.loewis.de>

> FYI, egg-info directories can store arbitrary data (see e.g. the
> EggTranslations package, which uses it for localization resources), so
> you may want to impose some restrictions on *which* metadata files to
> include.

You mean, so that it doesn't include malware, porn, or other spam?
That would be useful, I guess.

> Second, if a user uploads a source distribution built with setuptools,
> it will include an .egg-info directory automatically, so you can simply
> extract it from there. Conversely, if the source distribution is *not*
> built with setuptools, then building it with setuptools would not
> produce much information in the egg-info anyway.

I see. I'll try to start with that as an assumption, but will still try
to validate it wrt. real data.

> PKG-INFO
> SOURCES.txt # a manifest of the sdist contents
> top_level.txt # list of top-level package/module names
> scripts/ # source code of scripts in the package)
> zip-safe or not-zip-safe # flag file
>
> I don't think that most of these are useful for PyPI searches, though I
> suppose a listing of the name of the scripts the package includes could
> be useful.

I really try PyPI not to interpret any of these data. So I rather err on
the inclusive side.

> I'm not sure how useful it is to just have URLs for accessing the files,
> though, vs. having actual searches on structured data provided in the
> files. For example, an index of projects by package or module names, or
> of projects that provide a particular entry point.

In principle, it should be possible to let other services provide such
indices. I'd rather provide the files as-is for the moment, and see
what kind of facilities people would desire on top of it.

Regards,
Martin

From martin at v.loewis.de  Sat Sep 18 09:44:29 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 09:44:29 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
Message-ID: <4C946DDD.6080303@v.loewis.de>

> I'm confused, wouldn't that basically be a slap in the face for the
> people having worked on PEP345 and distutils2, especially during the
> Summer of Code?

How so? That work is still useful.

> Also, and I understand enthusiasm tends to build up during
> conferences, but wouldn't supporting setuptools' egg-info directory
> again be a step backwards after all those months of discussion about
> the direction of Python packaging?

People have been requesting this specific feature for years; until
now, I never understood what the need is. It's really a tiny feature
and very easy to provide. I fail to see why this can be a bad thing.

Regards,
Martin

From martin at v.loewis.de  Sat Sep 18 09:52:01 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 09:52:01 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
Message-ID: <4C946FA1.2030705@v.loewis.de>

> I am in full agreement with Tarek here. At ActiveState, we maintain
> our own index that differs from PyPI in two ways (among others):

I think you are saying something very different from what Tarek
says. IIUC, you are saying that egg-info is ill-defined and may
cause subtle problems. So you are saying there is a problem for
the users of the data. I could live with that - it's the user's
choice to use these data, after all.

Tarek is saying that it will be evil and bad for the community
to unpack some zip files. I find that statement itself counter-productive.

> Ideally, in future - I should be able to query static metadata (with
> environment markers[2] and such) for *any* package from PyPI. And
> this static metadata is simply a DIST-INFO file (instead of being a
> directory with a bunch of files in it). I don't really see a point in
> providing access to, say, the list of entry points of each package.

Again, that is completely different from what Tarek is saying.

You said (just now, and literally): "I don't really see a point".
Converting this to the -1/0/+1 system, it sounds like +0 to me:
you are unlikely to ever use this data.

This is perfectly fine with me: I won't use the data myself, either.
However, I fail to see why this should stop me from providing the
data, when there are people requesting this feature over and over
again. I'd like to see some of Python's "consenting adults" attitude
here.

Regards,
Martin

From fuzzyman at voidspace.org.uk  Sat Sep 18 11:44:27 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 18 Sep 2010 10:44:27 +0100
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C946DDD.6080303@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<4C946DDD.6080303@v.loewis.de>
Message-ID: <AANLkTinb+F-4ZrVa42wZLHdXu4ZWRHX8h3ppcrP-_1Wa@mail.gmail.com>

On 18 September 2010 08:44, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> [snip...]
>
>> Also, and I understand enthusiasm tends to build up during
>> conferences, but wouldn't supporting setuptools' egg-info directory
>> again be a step backwards after all those months of discussion about
>> the direction of Python packaging?
>>
>
> People have been requesting this specific feature for years; until
> now, I never understood what the need is. It's really a tiny feature
> and very easy to provide. I fail to see why this can be a bad thing.
>


Well, three years ago it might have been a good thing. :-)

With the distutils2 work very close to landing in the standard library,
providing infrastructure that will cause tools to *depend* on the old
formats sounds to me like a very bad idea. If tool use this metadata then it
could well prevent packages that want to be compatible with these tools from
using distutils2.

What PyPI does effectively becomes "the standard" for a large chunk of the
Python world (which is why changing the format PyPI provides data in can be
so hard). Now seems a really dumb time to bless the setuptools metadata
format as the defacto standard after so much work has gone into replacing it
and that effort is so close to completion.

All the best,

Michael Foord



>
> Regards,
> Martin
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>



-- 
http://www.voidspace.org.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100918/a2498b92/attachment-0001.html>

From martin at v.loewis.de  Sat Sep 18 12:03:58 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 12:03:58 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C948AEC.90607@voidspace.org.uk>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
Message-ID: <4C948E8E.7080403@v.loewis.de>

> With the distutils2 work very close to landing in the standard library,
> providing infrastructure that will cause tools to *depend* on the old
> formats is a very bad idea.

I think you are misunderstanding. The infrastructure will *not* depend
on the old formats. Instead, packaging that have that information will
provide it, packages that don't will not. The infrastructure is entirely
agnostic on whether the data is available or not. In particular, it will
not try to interpret the data in any way.

> If tool use this metadata then it could well
> prevent packages that want to be compatible with these tools from using
> distutils2.

I don't think this can well happen. In most known use cases, the tools
could support both forms of metadata. It may be that tools want to use
information that is not provided by PEP 345. However, the tool will then
likely fall back to just not having the information, as even existing
packages only provide that information occasionally.

> What PyPI does effectively becomes "the standard" for a large chunk of
> the Python world (which is why changing the format PyPI provides data in
> can be so hard). Now seems a really dumb time to bless the setuptools
> metadata format as the defacto standard after so much work has gone into
> replacing it and that effort is so close to completion.

Please understand that the information is not being "blessed"
(in any sense of the word that comes out of the dictionary). It's just
being made available slightly more conveniently - please understand that
it was available all the years already.

> So - I agree with Tarek. Exposing this information on PyPI would be
> problematic for the Python community. Not only does the data have the
> problems that Tarek and Sridhar point out, but it would actively hinder
> adoption of the better replacement.

That's really sad. So people will have to wait a few years to 
efficiently implement tools that they could implement today.

Regards,
Martin

From jannis at leidel.info  Sat Sep 18 13:11:44 2010
From: jannis at leidel.info (Jannis Leidel)
Date: Sat, 18 Sep 2010 13:11:44 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C946DDD.6080303@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<4C946DDD.6080303@v.loewis.de>
Message-ID: <1F5FB1D8-0010-43C5-A2E3-C72FDD0DA296@leidel.info>

On 18.09.2010, at 09:44, Martin v. L?wis wrote:

>> I'm confused, wouldn't that basically be a slap in the face for the
>> people having worked on PEP345 and distutils2, especially during the
>> Summer of Code?
> 
> How so? That work is still useful.

Well, useful like a toothless tiger. 

> Also, and I understand enthusiasm tends to build up during
>> conferences, but wouldn't supporting setuptools' egg-info directory
>> again be a step backwards after all those months of discussion about
>> the direction of Python packaging?
> 
> People have been requesting this specific feature for years; until
> now, I never understood what the need is. It's really a tiny feature
> and very easy to provide. I fail to see why this can be a bad thing.

Could a few of those "people" please provide us with concrete examples (in code) that'd make use of this data?

Jannis


From thomas at thomas-lotze.de  Sat Sep 18 11:29:32 2010
From: thomas at thomas-lotze.de (Thomas Lotze)
Date: Sat, 18 Sep 2010 11:29:32 +0200
Subject: [Catalog-sig] egg_info in PyPI
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
Message-ID: <pan.2010.09.18.09.29.31.664640@ID-174572.user.uni-berlin.de>

Hi there,

I'm going to add my own 2 cents to the discussion as I'm involved in the
matter here at the DZUG conference.

Tarek Ziad? wrote:

> Now you want to publish another metadata format at PyPI ? If PyPI takes
> that direction and adopts, promotes and publishes a standard that is not
> the one we worked on in the past year, this will increase our difficulty
> to push the new format so its adopted by the tools then the community.

Actually, I see it being the other way around: People use what used to
work for them and most people won't switch to using the new standard just
for the fun of it. In order for the new standard to acquire some momentum,
people need to be interested in the subject the standard is about and see
some concrete advantage in switching. Exposing existing metadata will make
it possible to build new applications that use metadata information; it's
those applications' responsibility then to promote the new standard.

Let me take the tl.eggdeps package as a concrete example. The package
currently analyses dependencies between installed packages. I'd like to
expand it to analyse dependencies between any packages on PyPI but I can't
as long as dependency information is not available without actually
installing things. Exposing PEP 345 metadata on PyPI won't be of any
practical use until a critical number of packages actually specify that
information. I simply won't bother implementing the feature either until
some point far in the future or unless I can use legacy metadata as a
fall-back. Having the legacy metadata available enables me to build the
feature now; this could raise interest in metadata as such, and I might
even make visually apparent which dependencies have been specified by PEP
345 means and which I could only guess from legacy information.

> People will just get confused because they will find two competing
> metadata formats  That's exactly the situation where we were at, and
> that's exactly where I don't want to go back.

That's simply a matter of documentation. It has to be clearly communicated
that the legacy information is provided as a transitional service and that
users of that metadata use it only until the newly specified information
is available for a critical number of packages (the exact value of which
may of course depend on the application in question).

> I am not even understanding what's the benefit of doing this since an
> egg_info directory is obtained at *build* time and can differ from a
> machine to another, so it seems pretty useless for me to publish this.

We do understand that legacy metadata has its issues, but taken with the
appropriate amount of salt, it's better than nothing in most cases, enough
so at least to solve some chicken-and-egg problems.

> We worked hard to build some standards, but if PyPI doesn't help us here,
> everything we did and are doing is useless.

Nothing that we're talking about in this thread will contradict what
you want to achieve and what work you've done. By the reasoning I tried to
explain above, what PyPI is going to do will actually help you. OTOH, I
personally don't think it would be a constructive way of helping you if
PyPI tried to enforce the new standard by keeping people from using
information they may at the moment only get through legacy means.

-- 
Thomas



From martin at v.loewis.de  Sat Sep 18 13:25:34 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 13:25:34 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C9496E8.1020405@voidspace.org.uk>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
Message-ID: <4C94A1AE.7010202@v.loewis.de>

>> I think you are misunderstanding. The infrastructure will *not* depend
>> on the old formats. Instead, packaging that have that information will
>> provide it, packages that don't will not. The infrastructure is entirely
>> agnostic on whether the data is available or not. In particular, it will
>> not try to interpret the data in any way.
>>
> No, not at all. If tools *use* the information (and if they don't then
> what is the point?) then packages that want to be compatible with those
> tools will have to provide this information.

Not true. Tools could/should also support PEP 345 data, and then they 
can support either kind of package.

> By providing information in this format PyPI will (like it or not) be
> blessing this format as the 'standard' way of providing this
> information.

By that definition, *both* formats are "blessed". The PEP 345 data
is already blessed. Depending on the definition of "providing", the 
egg-info data are also already "provided", ever since PyPI started
accepting egg files.

>> I don't think this can well happen. In most known use cases, the tools
>> could support both forms of metadata.
> Well, a) I would like to see that demonstrated

The tool in question is tl.eggdepend. It can easily support both kinds
of metadata.

> and b) having one
> standard is *far* preferable and having the distutils2 format be that
> standard is also far preferable. Please wait a bit (or start on
> supporting the distutils2 metadata format now).

The latter is already the case: the distutils2 metadata *is* supported
*now*. It's just that no package is using it (except for pep345demo).

As for a bit: how long exactly?

>> That's really sad. So people will have to wait a few years to
>> efficiently implement tools that they could implement today.
>
> Why a few years?

Because it will take that long until a significant number of
packages will use distutils 2. People still use very old versions
of packaging tools (e.g. the ones that come with Debian); it will
just take time to get the new tools and API adopted.

Regards,
Martin

From martin at v.loewis.de  Sat Sep 18 13:28:12 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 13:28:12 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <1F5FB1D8-0010-43C5-A2E3-C72FDD0DA296@leidel.info>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<4C946DDD.6080303@v.loewis.de>
	<1F5FB1D8-0010-43C5-A2E3-C72FDD0DA296@leidel.info>
Message-ID: <4C94A24C.2020209@v.loewis.de>

> Could a few of those "people" please provide us with concrete examples (in code) that'd make use of this data?

Unfortunately, I personally don't recall all the people that had been 
requesting something like this over the years. Currently, Thomas Lotze
and Jim Fulton have been supporting this request - hopefully, they'll
speak for themselves.

Regards,
Martin


From martin at v.loewis.de  Sat Sep 18 15:21:49 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 15:21:49 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94AF63.8010502@voidspace.org.uk>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk>
Message-ID: <4C94BCED.8010405@v.loewis.de>

> No. See above comment. If exposing this information has no value then
> don't do it. If it does have value, then we are blessing it - and
> therefore blessing it *over* other formats.

No: not *over*. Only over formats that don't get exposed. However,
the PEP 345 data are *already* exposed, via HTML, JSON, XML-RPC.
So they are much more prominently presented than what I'm proposing
to do. I fail to see why just extracting the egg-info would be
exposing it *over* the PEP 345 data.

>> The tool in question is tl.eggdepend. It can easily support both kinds
>> of metadata.
>>
> I couldn't find any references "tl.eggdepend" on the web. It is a minor
> point though.

Oops, see http://pypi.python.org/pypi/tl.eggdeps

> As in exported by PyPI though an API / interface?

Sure. See, for example,

http://pypi.python.org/pypi/pep345demo/json

It's also available through the XML-RPC release_data.

> Saying that they *could* is more empty words if
> our *actions* promote the use of another format.

But they do. Please stop saying that they might not, when they actually 
do (and have been for a while).

>> It's just that no package is using it (except for pep345demo).
>>
>> As for a bit: how long exactly?
> Distutils2 1a2 will be released in the next few days.

Sure. But when can tools computing dependencies for widely used packages
actually expect that these metadata will be available?

> Promoting another format in preference to distutils2 will very much
> prolong that.

IT WILL BE NOT IN PREFERENCE TO DISTUTILS2.

Regards,
Martin

From ziade.tarek at gmail.com  Sat Sep 18 16:58:06 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 18 Sep 2010 16:58:06 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94C63C.1060505@voidspace.org.uk>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk>
	<4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk>
Message-ID: <AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>

On Sat, Sep 18, 2010 at 4:01 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> ?Ok, I'm sorry - PEP 345 information is available via the PyPI API. (So
> exposing egg_info would not be promoting it *over* distutils2 but it would
> still be promoting and blessing it).
>
> Tarek's main point still stands though. The dependency information in the
> egg_info is tied to the platform and Python version that the package was
> *built* on. Neither pip nor easy_install use the egg_info to determine this;
> instead they re-generate it to get the correct information for the target
> platform.
>
> So if the use case is to provide dependency information exposing egg_info is
> not the right way to do it - and tools that use it will be using potentially
> (and frequently) inaccurate information. I stand by the point that once we
> start providing this information tools will start using it, and they
> shouldn't be. (So your question "when can tools computing dependencies for
> widely used packages actually expect that these metadata will be available?"
> is not answered by providing access to egg_info.)
>
> This problem (static availability of dependency information) is *exactly*
> the problem PEP 345 solves, so we should be pushing for progress on this
> front (distutils2a1 is already out and distutils2a2 will be out soon).

Sorry to take back the discussion late I was away for a while.

My main concern, like Michael says, is to see PyPI publish some
information that are known to be accurate only on one platform, the
one that is used to create the egg_info. Granted, most of the time the
information is the same no matter what the platform is, but still: you
don't know if this is the case when you read those metadata . The only
way to do it properly is to re-run egg_info.

So, right now Pip and easy_install are running egg_info to re-build
those data before they start the install process. They re-generate the
egg_info every time, and then browse the list of dependencies to get
them.

If you upload the egg_info data at PyPI so people can browse the
dependencies, you basically make the assumption that the egg_info
produced will work no matter what the platform is, unless you
associate with the egg_info the platform that was used.

So, I don't understand what is the benefit here, since a serious
installer will re-run egg_info every time.

Coming from the Plone/buildout community, I would be concerned if
buildout would rely on this. json, ldap, mysql you name it --I have
tons of other examples--  all those dependencies will not be accurate
unless you re-run setup.py egg_info. "Good enough metadata" sounds
completely wrong to me.

Cheers
Tarek

-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Sat Sep 18 17:10:46 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 17:10:46 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94C63C.1060505@voidspace.org.uk>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk>
	<4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk>
Message-ID: <4C94D676.8080703@v.loewis.de>

> So if the use case is to provide dependency information exposing
> egg_info is not the right way to do it - and tools that use it will be
> using potentially (and frequently) inaccurate information. I stand by
> the point that once we start providing this information tools will start
> using it, and they shouldn't be.

I think this is precisely the flaw in this line of reasoning: that we
refuse a service because we think the data is useless. Why can't the
users decide for themselves whether the data is useless?

As Thomas points out, people running into this will actually have a
reason to migrate to PEP 345. So exposing this presumed-incorrect
information will actually help promoting PEP 345.

> This problem (static availability of dependency information) is
> *exactly* the problem PEP 345 solves, so we should be pushing for
> progress on this front (distutils2a1 is already out and distutils2a2
> will be out soon).

And I don't mind pushing it, unless that means to deny users service
now based on the promise of offering better service tomorrow.

Regards,
Martin

From martin at v.loewis.de  Sat Sep 18 17:19:46 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 17:19:46 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>	<4C946FA1.2030705@v.loewis.de>	<4C948AEC.90607@voidspace.org.uk>	<4C948E8E.7080403@v.loewis.de>	<4C9496E8.1020405@voidspace.org.uk>	<4C94A1AE.7010202@v.loewis.de>	<4C94AF63.8010502@voidspace.org.uk>	<4C94BCED.8010405@v.loewis.de>	<4C94C63C.1060505@voidspace.org.uk>
	<AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>
Message-ID: <4C94D892.4070708@v.loewis.de>

> So, I don't understand what is the benefit here, since a serious
> installer will re-run egg_info every time.

I think the main applications that people are after are not builds.
They want the dependency information without downloading the packages,
and dependency information for packages they have no plans to install.

In the specific case of tl.eggdeps, the dependency information is only
used to create printable graphs. If this turns out to be slightly 
incorrect, people would notice if they try to use the packages in
question.

> Coming from the Plone/buildout community, I would be concerned if
> buildout would rely on this. json, ldap, mysql you name it --I have
> tons of other examples--  all those dependencies will not be accurate
> unless you re-run setup.py egg_info. "Good enough metadata" sounds
> completely wrong to me.

Still, people ask for it. I'm fine with telling them that the data
is flawed for various reasons. I object to denying them the data,
and I really dislike having to discard the patch that I already
wrote to implement it.

Regards,
Martin

From ziade.tarek at gmail.com  Sat Sep 18 17:27:33 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 18 Sep 2010 17:27:33 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94D892.4070708@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk>
	<4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk>
	<AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>
	<4C94D892.4070708@v.loewis.de>
Message-ID: <AANLkTimC27or8_TczZi_Zfyg7J4kSA9N8rAPYjXSHm-1@mail.gmail.com>

2010/9/18 "Martin v. L?wis" <martin at v.loewis.de>:
>> So, I don't understand what is the benefit here, since a serious
>> installer will re-run egg_info every time.
>
> I think the main applications that people are after are not builds.
> They want the dependency information without downloading the packages,
> and dependency information for packages they have no plans to install.

Yes they want to build the graph of dependencies, which will be
potentially false, as I explained.

>
> In the specific case of tl.eggdeps, the dependency information is only
> used to create printable graphs. If this turns out to be slightly incorrect,
> people would notice if they try to use the packages in
> question.

So you are fine with publishing "slightly incorrect" metadata at PyPI
? I am not.


>
>> Coming from the Plone/buildout community, I would be concerned if
>> buildout would rely on this. json, ldap, mysql you name it --I have
>> tons of other examples-- ?all those dependencies will not be accurate
>> unless you re-run setup.py egg_info. "Good enough metadata" sounds
>> completely wrong to me.
>
> Still, people ask for it. I'm fine with telling them that the data
> is flawed for various reasons. I object to denying them the data,
> and I really dislike having to discard the patch that I already
> wrote to implement it.

The can have those data, by downloading the tarball of the project and
running the egg_info command. They will in this case have accurate
data in fact. So right now you don't deny them the data, it's one
command away from them.

I don't understand the rational behind providing flawed data at PyPI.
I am -1 on that.

Regards
Tarek

>
> Regards,
> Martin
>



-- 
Tarek Ziad? | http://ziade.org

From pje at telecommunity.com  Sat Sep 18 17:49:44 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 18 Sep 2010 11:49:44 -0400
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94D892.4070708@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk>
	<4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk>
	<AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>
	<4C94D892.4070708@v.loewis.de>
Message-ID: <20100918154948.96C763A403D@sparrow.telecommunity.com>

At 05:19 PM 9/18/2010 +0200, Martin v. L?wis wrote:
>In the specific case of tl.eggdeps, the dependency information is only
>used to create printable graphs. If this turns out to be slightly 
>incorrect, people would notice if they try to use the packages in
>question.

By the way, just providing this information for .egg files and *not* 
for sdists would ensure accuracy of the metadata for that 
platform/python version.



From martin at v.loewis.de  Sat Sep 18 18:00:58 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 18:00:58 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <AANLkTimC27or8_TczZi_Zfyg7J4kSA9N8rAPYjXSHm-1@mail.gmail.com>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>	<4C946FA1.2030705@v.loewis.de>	<4C948AEC.90607@voidspace.org.uk>	<4C948E8E.7080403@v.loewis.de>	<4C9496E8.1020405@voidspace.org.uk>	<4C94A1AE.7010202@v.loewis.de>	<4C94AF63.8010502@voidspace.org.uk>	<4C94BCED.8010405@v.loewis.de>	<4C94C63C.1060505@voidspace.org.uk>	<AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>	<4C94D892.4070708@v.loewis.de>
	<AANLkTimC27or8_TczZi_Zfyg7J4kSA9N8rAPYjXSHm-1@mail.gmail.com>
Message-ID: <4C94E23A.7010709@v.loewis.de>

> So you are fine with publishing "slightly incorrect" metadata at PyPI
> ? I am not.

I really have no intuition for in how many cases the data will be 
incorrect. However, if users find that the data is incorrect for 
specific package, they ought to complain to the maintainer.

> I don't understand the rational behind providing flawed data at PyPI.
> I am -1 on that.

-1 sounds much better than vetoing it :-) Taking the votes together,
I currently arrive at

Tarek -1
Michael -1 (?)
Jannis -1 (?)

Jim +1
Thomas +1

Any other opinions?

Regards,
Martin

From martin at v.loewis.de  Sat Sep 18 18:06:40 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 18:06:40 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <20100918154948.96C763A403D@sparrow.telecommunity.com>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>	<4C946FA1.2030705@v.loewis.de>
	<4C948AEC.90607@voidspace.org.uk>	<4C948E8E.7080403@v.loewis.de>
	<4C9496E8.1020405@voidspace.org.uk>	<4C94A1AE.7010202@v.loewis.de>
	<4C94AF63.8010502@voidspace.org.uk>	<4C94BCED.8010405@v.loewis.de>
	<4C94C63C.1060505@voidspace.org.uk>	<AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>	<4C94D892.4070708@v.loewis.de>
	<20100918154948.96C763A403D@sparrow.telecommunity.com>
Message-ID: <4C94E390.7050109@v.loewis.de>

Am 18.09.10 17:49, schrieb P.J. Eby:
> At 05:19 PM 9/18/2010 +0200, Martin v. L?wis wrote:
>> In the specific case of tl.eggdeps, the dependency information is only
>> used to create printable graphs. If this turns out to be slightly
>> incorrect, people would notice if they try to use the packages in
>> question.
>
> By the way, just providing this information for .egg files and *not* for
> sdists would ensure accuracy of the metadata for that platform/python
> version.

True (I presume - unless there are also dependencies on the specific
OS version or system installation that may affect the metadata).

OTOH, I do think that the users asking for that prefer per-release
information, despite the limitations that this may have.

OTTH, if the concerns could be relieved if egg-info would provided
for all files that have it, I could provide that as well/instead.

Regards,
Martin

From pje at telecommunity.com  Sat Sep 18 18:47:33 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 18 Sep 2010 12:47:33 -0400
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94E390.7050109@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk>
	<4C94BCED.8010405@v.loewis.de> <4C94C63C.1060505@voidspace.org.uk>
	<AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>
	<4C94D892.4070708@v.loewis.de>
	<20100918154948.96C763A403D@sparrow.telecommunity.com>
	<4C94E390.7050109@v.loewis.de>
Message-ID: <20100918164736.DAC723A403D@sparrow.telecommunity.com>

At 06:06 PM 9/18/2010 +0200, Martin v. L?wis wrote:
>Am 18.09.10 17:49, schrieb P.J. Eby:
>>At 05:19 PM 9/18/2010 +0200, Martin v. L?wis wrote:
>>>In the specific case of tl.eggdeps, the dependency information is only
>>>used to create printable graphs. If this turns out to be slightly
>>>incorrect, people would notice if they try to use the packages in
>>>question.
>>
>>By the way, just providing this information for .egg files and *not* for
>>sdists would ensure accuracy of the metadata for that platform/python
>>version.
>
>True (I presume - unless there are also dependencies on the specific
>OS version or system installation that may affect the metadata).

No, because an egg's egg-info is what it is.  easy_install doesn't 
rebuild that information, so it is correct by 
definition.  ;-)  (Certainly, it is what it will be used for 
dependency information.)


>OTOH, I do think that the users asking for that prefer per-release
>information, despite the limitations that this may have.
>
>OTTH, if the concerns could be relieved if egg-info would provided
>for all files that have it, I could provide that as well/instead.

I am +0 on the idea myself, as I don't think the plan is quite enough 
to be able to provide a user-experience upgrade for use cases besides 
"make me a dependency graph without downloading the distributions themselves".

It certainly would be nice to be able to say to the user, "here are 
the things I will need to download in order to fulfill your request," 
but if you have to download individual files to get at that 
information, I'm not sure how much it helps vs. just downloading the files.


From cournape at gmail.com  Sat Sep 18 12:48:46 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 18 Sep 2010 19:48:46 +0900
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C9496E8.1020405@voidspace.org.uk>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
Message-ID: <AANLkTikDwntmYyKnRHjMV5u4BwUvL=LjzgqkOVAjHRQp@mail.gmail.com>

On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> ?On 18/09/2010 11:03, "Martin v. L?wis" wrote:

>> That's really sad. So people will have to wait a few years to efficiently
>> implement tools that they could implement today.
>
> Why a few years?

That's the time it will take for all packages to support distutils2 ?

cheers,

David

From cournape at gmail.com  Sat Sep 18 13:01:07 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 18 Sep 2010 20:01:07 +0900
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94995C.5090200@voidspace.org.uk>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<AANLkTikDwntmYyKnRHjMV5u4BwUvL=LjzgqkOVAjHRQp@mail.gmail.com>
	<4C94995C.5090200@voidspace.org.uk>
Message-ID: <AANLkTim5=hKYMYPhcM9hM0Fy7FxvX9EdDu2Hk5T21a=h@mail.gmail.com>

On Sat, Sep 18, 2010 at 7:50 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> ?On 18/09/2010 11:48, David Cournapeau wrote:
>>
>> On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord
>> <fuzzyman at voidspace.org.uk> ?wrote:
>>>
>>> ?On 18/09/2010 11:03, "Martin v. L?wis" wrote:
>>>>
>>>> That's really sad. So people will have to wait a few years to
>>>> efficiently
>>>> implement tools that they could implement today.
>>>
>>> Why a few years?
>>
>> That's the time it will take for all packages to support distutils2 ?
>
> Not "all packages" support setuptools.

Sure, but supporting setuptools was kind of possible for packages
relying heavily on distutils, even if it was not simple and fragile.
Distutils2 being incompatible API-wise with distutils, I am not sure
it will be as "easy" as with setuptools. It may be, but the only way
to know is to do it, and the incentive rather unclear. It means that
anyway, a lot of infrastructure will have to support both "standards"
for the time being.

cheers,

David

From fuzzyman at voidspace.org.uk  Sat Sep 18 11:48:28 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 18 Sep 2010 10:48:28 +0100
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C946FA1.2030705@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de>
Message-ID: <4C948AEC.90607@voidspace.org.uk>

  On 18/09/2010 08:52, "Martin v. L?wis" wrote:
>> I am in full agreement with Tarek here. At ActiveState, we maintain
>> our own index that differs from PyPI in two ways (among others):
>
> I think you are saying something very different from what Tarek
> says. IIUC, you are saying that egg-info is ill-defined and may
> cause subtle problems. So you are saying there is a problem for
> the users of the data. I could live with that - it's the user's
> choice to use these data, after all.
>
> Tarek is saying that it will be evil and bad for the community
> to unpack some zip files. I find that statement itself counter-productive.

With the distutils2 work very close to landing in the standard library, 
providing infrastructure that will cause tools to *depend* on the old 
formats is a very bad idea. If tool use this metadata then it could well 
prevent packages that want to be compatible with these tools from using 
distutils2.

What PyPI does effectively becomes "the standard" for a large chunk of 
the Python world (which is why changing the format PyPI provides data in 
can be so hard). Now seems a really dumb time to bless the setuptools 
metadata format as the defacto standard after so much work has gone into 
replacing it and that effort is so close to completion.

So - I agree with Tarek. Exposing this information on PyPI would be 
problematic for the Python community. Not only does the data have the 
problems that Tarek and Sridhar point out, but it would actively hinder 
adoption of the better replacement.

All the best,

Michael Foord

>
>
>> Ideally, in future - I should be able to query static metadata (with
>> environment markers[2] and such) for *any* package from PyPI. And
>> this static metadata is simply a DIST-INFO file (instead of being a
>> directory with a bunch of files in it). I don't really see a point in
>> providing access to, say, the list of entry points of each package.
>
> Again, that is completely different from what Tarek is saying.
>
> You said (just now, and literally): "I don't really see a point".
> Converting this to the -1/0/+1 system, it sounds like +0 to me:
> you are unlikely to ever use this data.
>
> This is perfectly fine with me: I won't use the data myself, either.
> However, I fail to see why this should stop me from providing the
> data, when there are people requesting this feature over and over
> again. I'd like to see some of Python's "consenting adults" attitude
> here.
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


-- 
http://www.ironpythoninaction.com/


From fuzzyman at voidspace.org.uk  Sat Sep 18 12:39:36 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 18 Sep 2010 11:39:36 +0100
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C948E8E.7080403@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de>
Message-ID: <4C9496E8.1020405@voidspace.org.uk>

  On 18/09/2010 11:03, "Martin v. L?wis" wrote:
>> With the distutils2 work very close to landing in the standard library,
>> providing infrastructure that will cause tools to *depend* on the old
>> formats is a very bad idea.
>
> I think you are misunderstanding. The infrastructure will *not* depend
> on the old formats. Instead, packaging that have that information will
> provide it, packages that don't will not. The infrastructure is entirely
> agnostic on whether the data is available or not. In particular, it will
> not try to interpret the data in any way.
>
No, not at all. If tools *use* the information (and if they don't then 
what is the point?) then packages that want to be compatible with those 
tools will have to provide this information. That will require those 
packages to use setuptools and prevent them using distutils2.

By providing information in this format PyPI will (like it or not) be 
blessing this format as the 'standard' way of providing this 
information. That I think is a very bad idea at this point in the 
evolution of Python packaging tools.

>> If tool use this metadata then it could well
>> prevent packages that want to be compatible with these tools from using
>> distutils2.
>
> I don't think this can well happen. In most known use cases, the tools
> could support both forms of metadata. 
Well, a) I would like to see that demonstrated and b) having one 
standard is *far* preferable and having the distutils2 format be that 
standard is also far preferable. Please wait a bit (or start on 
supporting the distutils2 metadata format now).

> It may be that tools want to use
> information that is not provided by PEP 345. However, the tool will then
> likely fall back to just not having the information, as even existing
> packages only provide that information occasionally.
>
>> What PyPI does effectively becomes "the standard" for a large chunk of
>> the Python world (which is why changing the format PyPI provides data in
>> can be so hard). Now seems a really dumb time to bless the setuptools
>> metadata format as the defacto standard after so much work has gone into
>> replacing it and that effort is so close to completion.
>
> Please understand that the information is not being "blessed"
> (in any sense of the word that comes out of the dictionary). It's just
> being made available slightly more conveniently - please understand that
> it was available all the years already.
>

I'm sorry but this is wrong. By providing this information about 
packages in the *standard* Python package repository it is very much 
(whether intentionally or not) blessing it as a standard.

>> So - I agree with Tarek. Exposing this information on PyPI would be
>> problematic for the Python community. Not only does the data have the
>> problems that Tarek and Sridhar point out, but it would actively hinder
>> adoption of the better replacement.
>
> That's really sad. So people will have to wait a few years to 
> efficiently implement tools that they could implement today.

Why a few years?

All the best,

Michael

>
> Regards,
> Martin


-- 
http://www.ironpythoninaction.com/


From fuzzyman at voidspace.org.uk  Sat Sep 18 12:50:04 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 18 Sep 2010 11:50:04 +0100
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <AANLkTikDwntmYyKnRHjMV5u4BwUvL=LjzgqkOVAjHRQp@mail.gmail.com>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>	<4C946FA1.2030705@v.loewis.de>	<4C948AEC.90607@voidspace.org.uk>	<4C948E8E.7080403@v.loewis.de>	<4C9496E8.1020405@voidspace.org.uk>
	<AANLkTikDwntmYyKnRHjMV5u4BwUvL=LjzgqkOVAjHRQp@mail.gmail.com>
Message-ID: <4C94995C.5090200@voidspace.org.uk>

  On 18/09/2010 11:48, David Cournapeau wrote:
> On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord
> <fuzzyman at voidspace.org.uk>  wrote:
>>   On 18/09/2010 11:03, "Martin v. L?wis" wrote:
>>> That's really sad. So people will have to wait a few years to efficiently
>>> implement tools that they could implement today.
>> Why a few years?
> That's the time it will take for all packages to support distutils2 ?
Not "all packages" support setuptools.

Michael

> cheers,
>
> David


-- 
http://www.ironpythoninaction.com/


From fuzzyman at voidspace.org.uk  Sat Sep 18 14:24:03 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 18 Sep 2010 13:24:03 +0100
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94A1AE.7010202@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de>
Message-ID: <4C94AF63.8010502@voidspace.org.uk>

  On 18/09/2010 12:25, "Martin v. L?wis" wrote:
>>> I think you are misunderstanding. The infrastructure will *not* depend
>>> on the old formats. Instead, packaging that have that information will
>>> provide it, packages that don't will not. The infrastructure is 
>>> entirely
>>> agnostic on whether the data is available or not. In particular, it 
>>> will
>>> not try to interpret the data in any way.
>>>
>> No, not at all. If tools *use* the information (and if they don't then
>> what is the point?) then packages that want to be compatible with those
>> tools will have to provide this information.
>
> Not true. Tools could/should also support PEP 345 data, and then they 
> can support either kind of package.
>
But you are proposing that PyPI will provide an interface / API for 
exposing the setuptools information. So tools that get metadata from 
PyPI in this way (and depend on it - as they will if you expose it) will 
have functionality that only works for packages that provide this 
information in this form.

Saying tools "should" work with other formats too is empty words.


>> By providing information in this format PyPI will (like it or not) be
>> blessing this format as the 'standard' way of providing this
>> information.
>
> By that definition, *both* formats are "blessed". The PEP 345 data
> is already blessed. Depending on the definition of "providing", the 
> egg-info data are also already "provided", ever since PyPI started
> accepting egg files.
>
No. See above comment. If exposing this information has no value then 
don't do it. If it does have value, then we are blessing it - and 
therefore blessing it *over* other formats. I accept this is not your 
intention. It *will* be the effect.

>>> I don't think this can well happen. In most known use cases, the tools
>>> could support both forms of metadata.
>> Well, a) I would like to see that demonstrated
>
> The tool in question is tl.eggdepend. It can easily support both kinds
> of metadata.
>
I couldn't find any references "tl.eggdepend" on the web. It is a minor 
point though.

>> and b) having one
>> standard is *far* preferable and having the distutils2 format be that
>> standard is also far preferable. Please wait a bit (or start on
>> supporting the distutils2 metadata format now).
>
> The latter is already the case: the distutils2 metadata *is* supported
> *now*. 

As in exported by PyPI though an API / interface? If not then again, 
irrelevant. Tools that use the new interface you are proposing *won't* 
use that information. Saying that they *could* is more empty words if 
our *actions* promote the use of another format.

> It's just that no package is using it (except for pep345demo).
>
> As for a bit: how long exactly?
Distutils2 1a2 will be released in the next few days.

>
>>> That's really sad. So people will have to wait a few years to
>>> efficiently implement tools that they could implement today.
>>
>> Why a few years?
>
> Because it will take that long until a significant number of
> packages will use distutils 2. People still use very old versions
> of packaging tools (e.g. the ones that come with Debian); it will
> just take time to get the new tools and API adopted.

Promoting another format in preference to distutils2 will very much 
prolong that.

Michael

>
> Regards,
> Martin


-- 
http://www.ironpythoninaction.com/


From fuzzyman at voidspace.org.uk  Sat Sep 18 16:01:32 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 18 Sep 2010 15:01:32 +0100
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94BCED.8010405@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk>
	<4C94BCED.8010405@v.loewis.de>
Message-ID: <4C94C63C.1060505@voidspace.org.uk>

  Ok, I'm sorry - PEP 345 information is available via the PyPI API. (So 
exposing egg_info would not be promoting it *over* distutils2 but it 
would still be promoting and blessing it).

Tarek's main point still stands though. The dependency information in 
the egg_info is tied to the platform and Python version that the package 
was *built* on. Neither pip nor easy_install use the egg_info to 
determine this; instead they re-generate it to get the correct 
information for the target platform.

So if the use case is to provide dependency information exposing 
egg_info is not the right way to do it - and tools that use it will be 
using potentially (and frequently) inaccurate information. I stand by 
the point that once we start providing this information tools will start 
using it, and they shouldn't be. (So your question "when can tools 
computing dependencies for widely used packages actually expect that 
these metadata will be available?" is not answered by providing access 
to egg_info.)

This problem (static availability of dependency information) is 
*exactly* the problem PEP 345 solves, so we should be pushing for 
progress on this front (distutils2a1 is already out and distutils2a2 
will be out soon).

All the best,

Michael Foord

On 18/09/2010 14:21, "Martin v. L?wis" wrote:
>> No. See above comment. If exposing this information has no value then
>> don't do it. If it does have value, then we are blessing it - and
>> therefore blessing it *over* other formats.
>
> No: not *over*. Only over formats that don't get exposed. However,
> the PEP 345 data are *already* exposed, via HTML, JSON, XML-RPC.
> So they are much more prominently presented than what I'm proposing
> to do. I fail to see why just extracting the egg-info would be
> exposing it *over* the PEP 345 data.
>
>>> The tool in question is tl.eggdepend. It can easily support both kinds
>>> of metadata.
>>>
>> I couldn't find any references "tl.eggdepend" on the web. It is a minor
>> point though.
>
> Oops, see http://pypi.python.org/pypi/tl.eggdeps
>
>> As in exported by PyPI though an API / interface?
>
> Sure. See, for example,
>
> http://pypi.python.org/pypi/pep345demo/json
>
> It's also available through the XML-RPC release_data.
>
>> Saying that they *could* is more empty words if
>> our *actions* promote the use of another format.
>
> But they do. Please stop saying that they might not, when they 
> actually do (and have been for a while).
>
>>> It's just that no package is using it (except for pep345demo).
>>>
>>> As for a bit: how long exactly?
>> Distutils2 1a2 will be released in the next few days.
>
> Sure. But when can tools computing dependencies for widely used packages
> actually expect that these metadata will be available?
>
>> Promoting another format in preference to distutils2 will very much
>> prolong that.
>
> IT WILL BE NOT IN PREFERENCE TO DISTUTILS2.
>
> Regards,
> Martin


-- 
http://www.ironpythoninaction.com/


From ncoghlan at gmail.com  Sat Sep 18 19:27:13 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 19 Sep 2010 03:27:13 +1000
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94AF63.8010502@voidspace.org.uk>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de> <4C948AEC.90607@voidspace.org.uk>
	<4C948E8E.7080403@v.loewis.de> <4C9496E8.1020405@voidspace.org.uk>
	<4C94A1AE.7010202@v.loewis.de> <4C94AF63.8010502@voidspace.org.uk>
Message-ID: <AANLkTim_PkieSGYncpK+g_p3tUqaD9juEmhxEmeKLTW0@mail.gmail.com>

On Sat, Sep 18, 2010 at 10:24 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> ?On 18/09/2010 12:25, "Martin v. L?wis" wrote:
>>>>
>>>> I think you are misunderstanding. The infrastructure will *not* depend
>>>> on the old formats. Instead, packaging that have that information will
>>>> provide it, packages that don't will not. The infrastructure is entirely
>>>> agnostic on whether the data is available or not. In particular, it will
>>>> not try to interpret the data in any way.
>>>>
>>> No, not at all. If tools *use* the information (and if they don't then
>>> what is the point?) then packages that want to be compatible with those
>>> tools will have to provide this information.
>>
>> Not true. Tools could/should also support PEP 345 data, and then they can
>> support either kind of package.
>>
> But you are proposing that PyPI will provide an interface / API for exposing
> the setuptools information. So tools that get metadata from PyPI in this way
> (and depend on it - as they will if you expose it) will have functionality
> that only works for packages that provide this information in this form.
>
> Saying tools "should" work with other formats too is empty words.

If the idea is solely to use legacy setuptools data as a fallback if
the actual PEP 345 data is unavailable, it sounds like it would be far
more robust to have *PyPI* do the fallback, implementing it correctly
*once*, rather than simply exposing the raw setuptools data (which
*will* lead to client applications that *only* understand the
setuptools data and can't understand packages that only provide PEP
345 compliant metadata).

Throwing the raw data at client applications and saying "here, you
figure it out" when they can already do that by downloading and
interrogating the packages directly seems likely to cause more
confusion than anything else.

So +1 to a "allow_fallback_metadata" flag in appropriate PyPI APIs, -1
on exposing the legacy data directly.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia

From fuzzyman at voidspace.org.uk  Sat Sep 18 20:47:08 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 18 Sep 2010 19:47:08 +0100
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <AANLkTim_PkieSGYncpK+g_p3tUqaD9juEmhxEmeKLTW0@mail.gmail.com>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>	<4C946FA1.2030705@v.loewis.de>	<4C948AEC.90607@voidspace.org.uk>	<4C948E8E.7080403@v.loewis.de>	<4C9496E8.1020405@voidspace.org.uk>	<4C94A1AE.7010202@v.loewis.de>	<4C94AF63.8010502@voidspace.org.uk>
	<AANLkTim_PkieSGYncpK+g_p3tUqaD9juEmhxEmeKLTW0@mail.gmail.com>
Message-ID: <4C95092C.9020103@voidspace.org.uk>

  On 18/09/2010 18:27, Nick Coghlan wrote:
> On Sat, Sep 18, 2010 at 10:24 PM, Michael Foord
> <fuzzyman at voidspace.org.uk>  wrote:
>>   On 18/09/2010 12:25, "Martin v. L?wis" wrote:
>>>>> I think you are misunderstanding. The infrastructure will *not* depend
>>>>> on the old formats. Instead, packaging that have that information will
>>>>> provide it, packages that don't will not. The infrastructure is entirely
>>>>> agnostic on whether the data is available or not. In particular, it will
>>>>> not try to interpret the data in any way.
>>>>>
>>>> No, not at all. If tools *use* the information (and if they don't then
>>>> what is the point?) then packages that want to be compatible with those
>>>> tools will have to provide this information.
>>> Not true. Tools could/should also support PEP 345 data, and then they can
>>> support either kind of package.
>>>
>> But you are proposing that PyPI will provide an interface / API for exposing
>> the setuptools information. So tools that get metadata from PyPI in this way
>> (and depend on it - as they will if you expose it) will have functionality
>> that only works for packages that provide this information in this form.
>>
>> Saying tools "should" work with other formats too is empty words.
> If the idea is solely to use legacy setuptools data as a fallback if
> the actual PEP 345 data is unavailable, it sounds like it would be far
> more robust to have *PyPI* do the fallback, implementing it correctly
> *once*, rather than simply exposing the raw setuptools data (which
> *will* lead to client applications that *only* understand the
> setuptools data and can't understand packages that only provide PEP
> 345 compliant metadata).
>
> Throwing the raw data at client applications and saying "here, you
> figure it out" when they can already do that by downloading and
> interrogating the packages directly seems likely to cause more
> confusion than anything else.
>
> So +1 to a "allow_fallback_metadata" flag in appropriate PyPI APIs, -1
> on exposing the legacy data directly.
>

If the two different data formats can be exposed in a compatible way 
then this sounds good to me.

Michael

> Cheers,
> Nick.
>


-- 
http://www.ironpythoninaction.com/


From fdrake at acm.org  Sat Sep 18 21:47:46 2010
From: fdrake at acm.org (Fred Drake)
Date: Sat, 18 Sep 2010 15:47:46 -0400
Subject: [Catalog-sig] [Python-Dev] egg_info in PyPI
In-Reply-To: <4C94E23A.7010709@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>
	<4C946FA1.2030705@v.loewis.de>
	<4C948AEC.90607@voidspace.org.uk> <4C948E8E.7080403@v.loewis.de>
	<4C9496E8.1020405@voidspace.org.uk> <4C94A1AE.7010202@v.loewis.de>
	<4C94AF63.8010502@voidspace.org.uk> <4C94BCED.8010405@v.loewis.de>
	<4C94C63C.1060505@voidspace.org.uk>
	<AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>
	<4C94D892.4070708@v.loewis.de>
	<AANLkTimC27or8_TczZi_Zfyg7J4kSA9N8rAPYjXSHm-1@mail.gmail.com>
	<4C94E23A.7010709@v.loewis.de>
Message-ID: <AANLkTik-Q8CPoV4j-CLT0c3U-HOgQhTNWAYrDe7QgkAy@mail.gmail.com>

2010/9/18 "Martin v. L?wis" <martin at v.loewis.de>:
> Any other opinions?

-1 from me as well; I see no reason to encourage use of bad metadata,
given mechanisms to get correct metadata exist (running "setup.py
egg_info", as others have pointed out).  I understand there are
perceived uses for such data, but it's just as available for those
uses as for tools like pip and buildout.


? -Fred

--
Fred L. Drake, Jr.? ? <fdrake at acm.org>
"A storm broke loose in my mind."? --Albert Einstein

From mal at egenix.com  Sat Sep 18 23:09:35 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 18 Sep 2010 23:09:35 +0200
Subject: [Catalog-sig] [Python-Dev]  egg_info in PyPI
In-Reply-To: <4C94E23A.7010709@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>	<4C946FA1.2030705@v.loewis.de>	<4C948AEC.90607@voidspace.org.uk>	<4C948E8E.7080403@v.loewis.de>	<4C9496E8.1020405@voidspace.org.uk>	<4C94A1AE.7010202@v.loewis.de>	<4C94AF63.8010502@voidspace.org.uk>	<4C94BCED.8010405@v.loewis.de>	<4C94C63C.1060505@voidspace.org.uk>	<AANLkTimWdL4zeb_uKAyYQqqsJsuqe8JYtnz6q1G_uM3H@mail.gmail.com>	<4C94D892.4070708@v.loewis.de>	<AANLkTimC27or8_TczZi_Zfyg7J4kSA9N8rAPYjXSHm-1@mail.gmail.com>
	<4C94E23A.7010709@v.loewis.de>
Message-ID: <4C952A8F.9020202@egenix.com>

"Martin v. L?wis" wrote:
>> So you are fine with publishing "slightly incorrect" metadata at PyPI
>> ? I am not.
> 
> I really have no intuition for in how many cases the data will be
> incorrect. However, if users find that the data is incorrect for
> specific package, they ought to complain to the maintainer.
> 
>> I don't understand the rational behind providing flawed data at PyPI.
>> I am -1 on that.
> 
> -1 sounds much better than vetoing it :-) Taking the votes together,
> I currently arrive at
> 
> Tarek -1
> Michael -1 (?)
> Jannis -1 (?)

> Jim +1
> Thomas +1
> 
> Any other opinions?

-1 as well. The formats and file-names are just a complete mess and
not all packages on PyPI are available as eggs or compatible to
setuptools.

I think the information from PEP 345 which is already made
available via the XML-RPC .release_data() interface is sufficient
for most cases. In those cases where it is not, we should extend the
meta data.

http://www.python.org/dev/peps/pep-0345/
http://wiki.python.org/moin/PyPiXmlRpc

It may be worthwhile providing the same sort of information as static
PKG-INFO file alongside the /simple index entries as well (e.g.
via a link on the package page), to make it possible to extract
the meta information without having to use the RPC mechanism
and making use of the mirror infrastructure.

http://www.python.org/dev/peps/pep-0314/
http://www.python.org/dev/peps/pep-0390/

PS: Could you put your DZUG talk up online somewhere. It includes
some very valuable information on the various available APIs that
is difficult to find elswhere.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 18 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From martin at v.loewis.de  Sat Sep 18 23:15:04 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Sep 2010 23:15:04 +0200
Subject: [Catalog-sig] [Python-Dev] egg_info in PyPI
In-Reply-To: <i72eo2$nfs$1@dough.gmane.org>
References: <4C93B6D8.1090700@v.loewis.de>	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>	<6871B7C0-6C6D-4FA9-9709-D3AF996322BE@activestate.com>	<4C946FA1.2030705@v.loewis.de>	<4C948AEC.90607@voidspace.org.uk>	<4C948E8E.7080403@v.loewis.de>	<4C9496E8.1020405@voidspace.org.uk>	<4C94A1AE.7010202@v.loewis.de>	<4C94AF63.8010502@voidspace.org.uk>
	<4C94BCED.8010405@v.loewis.de> <i72eo2$nfs$1@dough.gmane.org>
Message-ID: <4C952BD8.6020701@v.loewis.de>

Am 18.09.2010 15:27, schrieb Steve Holden:
> On 9/18/2010 9:21 AM, "Martin v. L?wis" wrote:
>> IT WILL BE NOT IN PREFERENCE TO DISTUTILS2.
> 
> No need to shout.

I really felt that this otherwise wouldn't be heard - I tried
to say it a number of times before, and it was ignored.

Regards,
Martin

From sridharr at activestate.com  Sun Sep 19 03:40:42 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Sat, 18 Sep 2010 18:40:42 -0700
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <pan.2010.09.18.09.29.31.664640@ID-174572.user.uni-berlin.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<pan.2010.09.18.09.29.31.664640@ID-174572.user.uni-berlin.de>
Message-ID: <7171CCFF-5EF7-41E1-AD08-12BA0B0896FB@activestate.com>


On 2010-09-18, at 2:29 AM, Thomas Lotze wrote:

> I'd like to expand [tl.eggdeps]
> to analyse dependencies between any packages on PyPI but I can't
> as long as dependency information is not available without actually
> installing things. [...]

On 2010-09-18, at 2:29 AM, Thomas Lotze wrote:

>> I am not even understanding what's the benefit of doing this since an
>> egg_info directory is obtained at *build* time and can differ from a
>> machine to another, so it seems pretty useless for me to publish this.
> 
> We do understand that legacy metadata has its issues, but taken with the
> appropriate amount of salt, it's better than nothing in most cases, enough
> so at least to solve some chicken-and-egg problems.

Just curious: aside from the static metadata issue which you recognize, how would you make tl.eggdeps analyze dependencies between any packages registered on PyPI  when not all of them upload their sdists (hence no egg_info) to PyPI?

Also, if I may ask, what is the intended use for tl.eggdeps? For use in package managers (buildout, pip)? If yes, then ... would providing egg_info in PyPI obviate the need for package managers such as pip to download sources before resolving dependencies (keeping in mind that even packages that do upload their sdists to PyPI sometimes end up not doing the same - perhaps out of forgetfulness - for theirs latest releases)?

-srid

From thomas at thomas-lotze.de  Sun Sep 19 16:06:54 2010
From: thomas at thomas-lotze.de (Thomas Lotze)
Date: Sun, 19 Sep 2010 16:06:54 +0200
Subject: [Catalog-sig] egg_info in PyPI
References: <4C93B6D8.1090700@v.loewis.de>
	<1C681A95-392E-4001-B3F3-6131696EEF7C@leidel.info>
	<AANLkTimpcB6HB6WBjcq35wfGpRVGG_sn9+ocG_z7SuTv@mail.gmail.com>
	<pan.2010.09.18.09.29.31.664640@ID-174572.user.uni-berlin.de>
	<7171CCFF-5EF7-41E1-AD08-12BA0B0896FB@activestate.com>
Message-ID: <pan.2010.09.19.14.06.53.483481@ID-174572.user.uni-berlin.de>

Sridhar Ratnakumar wrote:

> Just curious: aside from the static metadata issue which you recognize,
> how would you make tl.eggdeps analyze dependencies between any packages
> registered on PyPI  when not all of them upload their sdists (hence no
> egg_info) to PyPI?

For packages that aren't even eggs, I can do nothing. For those which are
but just don't have any distribution files uploaded, we've considered
running a service (although most likely not on the PyPI server itself)
that tries to hunt down sdists or eggs the same way setuptools does.
Again, there's a number of packages with metadata available that I
consider good enough for use cases such as mine.

> Also, if I may ask, what is the intended use for tl.eggdeps?

One example is its use by the Zope project in analysing and cleaning up
dependencies between the packages in the zope namespace after Zope's
eggification.

> For use in
> package managers (buildout, pip)? If yes, then ... would providing
> egg_info in PyPI obviate the need for package managers such as pip to
> download sources before resolving dependencies

I'm not sure that it would be useful to share the dependency resolution
code between package managers and clients such as tl.eggdeps, and it is
certainly not my current intent. Maybe it turns out one day that a general
client library for consumers of dependency metadata is worthwhile, but I
imagine that different applications do widely varying things with the
dependency graph so who knows.

> (keeping in mind that even
> packages that do upload their sdists to PyPI sometimes end up not doing
> the same - perhaps out of forgetfulness - for theirs latest releases)?

Well, if a package doesn't make distributions or metadata for its latest
releases available, then all consumers of that information are out of luck
in the same way, aren't they?

-- 
Thomas




From martin at v.loewis.de  Mon Sep 20 11:09:06 2010
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 20 Sep 2010 11:09:06 +0200
Subject: [Catalog-sig] PyPI mirror selection
Message-ID: <4C9724B2.9060904@v.loewis.de>

I started a library that tools can use to select a PyPI mirror, see

https://svn.python.org/packages/trunk/pypi/tools/mirrorlib.py

It currently only deals with mirror selection, but will be extended
to deal with mirror validation and key rollover as well.

For mirror selection, the objective is to find a mirror that is both
fast and current.

1. The caller specifies a maximum acceptable response time
   for an HTTP request (to /last-modified), and a maximum acceptable
   age. The caller can also specify whether a.pypi.python.org should
   be included in the scan or not. Finally, the caller can specify
   a timeout for slow mirrors.

2. The library contacts the mirrors in order, interleaving DNS lookups
   with connecting to the mirrors whose IP addresses have been computed.
   No threads are created in that process.

3. If a mirror is found that meets the requirements, it is returned;
   this might mean that not all mirrors have been contacted.

4. If no mirror is found that meets the requirements, it contacts
   all mirrors. When the slow-mirrors timeout has passed, the youngest
   of all responding mirrors is returned.

5. If all mirrors are slow, the first one responding is returned.

6. If none respond, ValueError is raised (which will happen after
   the TCP connection timeout).

For specific parameters, I found the following defaults useful:
- the acceptable mirror age defaults to 30min. Within this time,
  all mirrors should have synchronized, otherwise, they are
  considered down. The only exception is when the central mirror
  is down, then the mirrors will all age.
- the acceptable response time defaults to 1s - all mirrors
  should be able to respond within this time, and it will then
  use the one that responds first, in enumeration order.
  Specifying 0.1s might also be useful in some applications;
  this will rule out slower mirrors (in particular, GAE).
- the slow mirrors timeout defaults to 5s. If the master is
  down, and some mirror is slow, this will be the time until
  selection completes (with the mirror that claims to have the
  latest copy).
- OTOH, if the master is down, and all mirrors respond to the
  TCP connect quickly (either accepting or refusing the connection),
  then it will quickly pick the newest mirror.

If there are any questions, feel free to ask.

Regards,
Martin



From merwok at netwok.org  Mon Sep 20 18:03:41 2010
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Mon, 20 Sep 2010 18:03:41 +0200
Subject: [Catalog-sig] PyPI mirror selection
In-Reply-To: <4C9724B2.9060904@v.loewis.de>
References: <4C9724B2.9060904@v.loewis.de>
Message-ID: <4C9785DD.6030506@netwok.org>

Hello

> I started a library that tools can use to select a PyPI mirror [...]
Nice, thank you.

> If there are any questions, feel free to ask.
The obvious question from a distutils person: Any plans to package the
lib and publish it on PyPI?  (Maybe with a more specific name.)

It seems very few edits would be needed to make the lib compatible with
3.x.  What is the minimum 2.x version you want to support?  How to send
patches?

Regards



From jim at zope.com  Mon Sep 20 20:38:48 2010
From: jim at zope.com (Jim Fulton)
Date: Mon, 20 Sep 2010 14:38:48 -0400
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C93B6D8.1090700@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
Message-ID: <AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>

Some high-level meta comments on this thread:

1. I broke one of my own rules. I should have asked: what problem is
   the proposed change intended to solve?  That way, even if the
   proposal wasn't right, people could have suggested better
   alternatives. :) My bad.

2. I'm pretty sure that the people who suggested making egg_info
   available didn't intend to slap anyone in the face or run afoul of
   current efforts. :)

   (Again, it would have been better to state the
   problem being solved and then request input on the proposed
   solution.)

Jim

--
Jim Fulton

From mal at egenix.com  Mon Sep 20 20:47:59 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 20 Sep 2010 20:47:59 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>
Message-ID: <4C97AC5F.90005@egenix.com>

Jim Fulton wrote:
> Some high-level meta comments on this thread:
> 
> 1. I broke one of my own rules. I should have asked: what problem is
>    the proposed change intended to solve?  That way, even if the
>    proposal wasn't right, people could have suggested better
>    alternatives. :) My bad.
> 
> 2. I'm pretty sure that the people who suggested making egg_info
>    available didn't intend to slap anyone in the face or run afoul of
>    current efforts. :)
> 
>    (Again, it would have been better to state the
>    problem being solved and then request input on the proposed
>    solution.)

I think the main problem that's been discussed over and over
again in recent years is the problem of reading package
meta-data without actually downloading the packages.

IMHO, most of this has been resolved by means of the XML-RPC
interface via the .release_data() method:

http://wiki.python.org/moin/PyPiXmlRpc

If there's something missing, we should add it there.

Additionally, it would make sense to have this data around
as static file for the PyPI mirrors to pick up. I suggested
adding package PKG-INFO files to the package pages available
on the /simple index for that, but other solutions may be better.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 20 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From martin at v.loewis.de  Mon Sep 20 21:22:34 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 20 Sep 2010 21:22:34 +0200
Subject: [Catalog-sig] PyPI mirror selection
In-Reply-To: <4C9785DD.6030506@netwok.org>
References: <4C9724B2.9060904@v.loewis.de> <4C9785DD.6030506@netwok.org>
Message-ID: <4C97B47A.7060409@v.loewis.de>

> The obvious question from a distutils person: Any plans to package the
> lib and publish it on PyPI?  (Maybe with a more specific name.)

Sure, will do (but I'd like to implement the missing features first).

> It seems very few edits would be needed to make the lib compatible with
> 3.x.  What is the minimum 2.x version you want to support?  How to send
> patches?

For the moment: svn diffs, by personal email to me.

Regards,
Martin

From martin at v.loewis.de  Mon Sep 20 21:26:31 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 20 Sep 2010 21:26:31 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C97AC5F.90005@egenix.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>
	<4C97AC5F.90005@egenix.com>
Message-ID: <4C97B567.5070104@v.loewis.de>

> IMHO, most of this has been resolved by means of the XML-RPC
> interface via the .release_data() method:
> 
> http://wiki.python.org/moin/PyPiXmlRpc
> 
> If there's something missing, we should add it there.

Very clearly, most of egg-info is missing there. However, I
sense that people are very opposed to adding it there.

I merely asked whether it was ok to expose the egg-info
data at all, and was met with strong opposition - nobody
actually asked what specific API to expose them I had in
mind.

> Additionally, it would make sense to have this data around
> as static file for the PyPI mirrors to pick up.

That's how I would have implemented it. Alas, this very
idea was just shot down.

Regards,
Martin

From mal at egenix.com  Mon Sep 20 21:57:52 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 20 Sep 2010 21:57:52 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C97B567.5070104@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>	<4C97AC5F.90005@egenix.com>
	<4C97B567.5070104@v.loewis.de>
Message-ID: <4C97BCC0.5000403@egenix.com>

"Martin v. L?wis" wrote:
>> IMHO, most of this has been resolved by means of the XML-RPC
>> interface via the .release_data() method:
>>
>> http://wiki.python.org/moin/PyPiXmlRpc
>>
>> If there's something missing, we should add it there.
> 
> Very clearly, most of egg-info is missing there. However, I
> sense that people are very opposed to adding it there.
> 
> I merely asked whether it was ok to expose the egg-info
> data at all, and was met with strong opposition - nobody
> actually asked what specific API to expose them I had in
> mind.

Well, "most" depends on what you view as really important
meta-data.

Here's an example EGG-INFO dir:

dependency_links.txt

 This file lists extra links to search for dependencies. This could
 be added to the meta-data.

native_libs.txt

 This is not really needed, since I think we all agree that
 keeping eggs zipped up when installing them is not a good idea.

not-zip-safe

 Ditto.

PKG-INFO

 This is basically what we already have in .release_data().

SOURCES.txt

 I don't know the purpose of this file. It's a manifest of
 source files. The setuptools docs say it's not used anywhere.

top_level.txt

 This file just lists the top-level package names made available
 by the package. We could add a meta-data field for this
 information or use the "Provides:" field for it.

namespace_packages.txt

 This file lists the namespace packages used by the package. We
 could add a meta-data field for this or just use the "Requires:"
 field for it.

>> Additionally, it would make sense to have this data around
>> as static file for the PyPI mirrors to pick up.
> 
> That's how I would have implemented it. Alas, this very
> idea was just shot down.

I think most of the response was targeted at the EGG-INFO
meta-data format, not the information itself.

If you look at the above mess of upper- and lower-case filenames,
files which only serve as marker and others having actual
content, different content types, etc. I think it's clear
that using this particular format is not a good idea.

We can do better :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 20 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Mon Sep 20 22:18:41 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 20 Sep 2010 22:18:41 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C97BCC0.5000403@egenix.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>
	<4C97AC5F.90005@egenix.com> <4C97B567.5070104@v.loewis.de>
	<4C97BCC0.5000403@egenix.com>
Message-ID: <AANLkTinYq98fZF+uM=uWspxVmW-0fct==oHfah_bxgB+@mail.gmail.com>

On Mon, Sep 20, 2010 at 9:57 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> That's how I would have implemented it. Alas, this very
>> idea was just shot down.
>
> I think most of the response was targeted at the EGG-INFO
> meta-data format, not the information itself.

No, it's also the information itself. requires.txt for instance can
vary depending on the platform that was initially used to build
egg-info. IOW it means that you push data that is created at build
time at PyPI. Publishing those files and calling them metadata is
inaccurate.

So the status-quo where people need to rebuild egg_info to get the
metadata for their platform is preferable, until PEP 345 gets used,
since is was partially created to solve this.

Regards
Tarel

From ziade.tarek at gmail.com  Mon Sep 20 22:25:17 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 20 Sep 2010 22:25:17 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C97B567.5070104@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>
	<4C97AC5F.90005@egenix.com> <4C97B567.5070104@v.loewis.de>
Message-ID: <AANLkTimXr2iiz+DBMx2hiEQuhQoNhAeM5Rix4YebdKaa@mail.gmail.com>

On Mon, Sep 20, 2010 at 9:26 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> IMHO, most of this has been resolved by means of the XML-RPC
>> interface via the .release_data() method:
>>
>> http://wiki.python.org/moin/PyPiXmlRpc
>>
>> If there's something missing, we should add it there.
>
> Very clearly, most of egg-info is missing there. However, I
> sense that people are very opposed to adding it there.
>
> I merely asked whether it was ok to expose the egg-info
> data at all, and was met with strong opposition - nobody
> actually asked what specific API to expose them I had in
> mind.

If the use case is to provide a way to build a dependency graph just
by querying PyPI, that's exactly what we have in mind in pep 345.

Now, the strong opposition is because you are proposing something that
doesn't work in all cases as I mentioned, and that can be done
accurately by downloading the package then calling egg_info, like how
pip does for installing a package. Which is I think good enough until
pep 345 is used. Slower, but accurate.

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Mon Sep 20 22:16:43 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 20 Sep 2010 22:16:43 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C97BCC0.5000403@egenix.com>
References: <4C93B6D8.1090700@v.loewis.de>	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>	<4C97AC5F.90005@egenix.com>
	<4C97B567.5070104@v.loewis.de> <4C97BCC0.5000403@egenix.com>
Message-ID: <4C97C12B.9070106@v.loewis.de>

> I think most of the response was targeted at the EGG-INFO
> meta-data format, not the information itself.

My impression was otherwise: several posters clearly indicated
that they don't want exposition of this information (in whatever
form), as they expected it would hinder adoption of PEP 345.

> If you look at the above mess of upper- and lower-case filenames,
> files which only serve as marker and others having actual
> content, different content types, etc. I think it's clear
> that using this particular format is not a good idea.
> 
> We can do better :-)

Unfortunately, it seems, not with the data available.

Regards,
Martin

From mal at egenix.com  Mon Sep 20 22:34:20 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 20 Sep 2010 22:34:20 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <AANLkTimXr2iiz+DBMx2hiEQuhQoNhAeM5Rix4YebdKaa@mail.gmail.com>
References: <4C93B6D8.1090700@v.loewis.de>	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>	<4C97AC5F.90005@egenix.com>
	<4C97B567.5070104@v.loewis.de>
	<AANLkTimXr2iiz+DBMx2hiEQuhQoNhAeM5Rix4YebdKaa@mail.gmail.com>
Message-ID: <4C97C54C.9050405@egenix.com>

Tarek Ziad? wrote:
> On Mon, Sep 20, 2010 at 9:26 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>> IMHO, most of this has been resolved by means of the XML-RPC
>>> interface via the .release_data() method:
>>>
>>> http://wiki.python.org/moin/PyPiXmlRpc
>>>
>>> If there's something missing, we should add it there.
>>
>> Very clearly, most of egg-info is missing there. However, I
>> sense that people are very opposed to adding it there.
>>
>> I merely asked whether it was ok to expose the egg-info
>> data at all, and was met with strong opposition - nobody
>> actually asked what specific API to expose them I had in
>> mind.
> 
> If the use case is to provide a way to build a dependency graph just
> by querying PyPI, that's exactly what we have in mind in pep 345.

I think I lost you there... .release_data() does provide the
PEP 345 meta-data, at least according to the documentation.

Whether PyPI actually has the data depends on how the release
was registered with PyPI, of course.

BTW: Something in the RPC interface or the distutils
register command is not really up to speed yet, since the
data appears to be a bit mangled at the moment (key and value
are reversed for a few entries):

>>> pprint.pprint(client.release_data('egenix-mxodbc', '3.1.0'))
{'3.1.0': 'cheesecake_documentation_id',
 'Windows,Linux,FreeBSD,Solaris,Mac OS X,AIX': 'version',
 '_pypi_hidden': <Boolean False at 933488>,
 '_pypi_ordering': 15,
 'classifiers': ['Development Status :: 5 - Production/Stable',
                 'Development Status :: 6 - Mature',
                 'Environment :: Console',
                 'Environment :: No Input/Output (Daemon)',
                 'Intended Audience :: Developers',
                 'License :: Other/Proprietary License',
                 'Natural Language :: English',
                 'Operating System :: MacOS',
                 'Operating System :: Microsoft :: Windows',
                 'Operating System :: OS Independent',
                 'Operating System :: Other OS',
                 'Operating System :: POSIX',
                 'Operating System :: Unix',
                 'Programming Language :: C',
                 'Programming Language :: Python',
                 'Programming Language :: Python :: 2',
                 'Programming Language :: Python :: 2.3',
                 'Programming Language :: Python :: 2.4',
                 'Programming Language :: Python :: 2.5',
                 'Programming Language :: Python :: 2.6',
                 'Programming Language :: Python :: 2.7',
                 'Topic :: Communications',
                 'Topic :: Database',
                 'Topic :: Database :: Database Engines/Servers',
                 'Topic :: Database :: Front-Ends',
                 'Topic :: Documentation',
                 'Topic :: Internet',
                 'Topic :: Office/Business',
                 'Topic :: Software Development',
                 'Topic :: Software Development :: Libraries',
                 'Topic :: Software Development :: Libraries :: Application Frameworks',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 'Topic :: Utilities'],
 'description': 'The mxODBC Database Interface for Python is a commercial add-on to our\nopen-source
eGenix mx Extension Series, providing robust and proven database\naccess for Python on all major
platforms, such as Windows, Linux, Mac OS X, \nFreeBSD, Solaris, using a single API.\n\nmxODBC works
with Python 2.3 - 2.7 and supports 32-bit as well as\n64-bit platforms.\n\nPlease contact
sales at egenix.com for evaluation licenses or visit the\neGenix.com online-shop to purchase licenses
at http://www.egenix.com/.\n\nThis software is brought to you by eGenix.com and distributed
under\nthe eGenix.com Commercial License 1.3.0.',
 'eGenix.com GmbH': 'author_email',
 'home_page': 'http://www.egenix.com/products/python/mxODBC/',
 'http://pypi.python.org/pypi/egenix-mxodbc': 'author',
 'http://www.egenix.com/products/python/mxODBC/': 'platform',
 'info at egenix.com': 'download_url',
 'keywords': 'package_url',
 'license': 'eGenix.com Commercial License 1.3.0; Copyright (c) 1997-2000, Marc-Andre Lemburg, All
Rights Reserved; Copyright (c) 2000-2010, eGenix.com Software GmbH, All Rights Reserved',
 'maintainer': 'requires_python',
 'maintainer_email': 'cheesecake_code_kwalitee_id',
 'name': 'egenix-mxodbc',
 'release_url': 'http://pypi.python.org/pypi/egenix-mxodbc/3.1.0',
 'stable_version': 'cheesecake_installability_id',
 'summary': 'eGenix mxODBC - ODBC Database Interface for Python'}

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 20 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Mon Sep 20 22:47:28 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 20 Sep 2010 22:47:28 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C97C54C.9050405@egenix.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>
	<4C97AC5F.90005@egenix.com> <4C97B567.5070104@v.loewis.de>
	<AANLkTimXr2iiz+DBMx2hiEQuhQoNhAeM5Rix4YebdKaa@mail.gmail.com>
	<4C97C54C.9050405@egenix.com>
Message-ID: <AANLkTikwatQgN3E4_Ar8niYOpp+8KxYRvg8EUXk6qkNm@mail.gmail.com>

2010/9/20 M.-A. Lemburg <mal at egenix.com>:
> Tarek Ziad? wrote:
>> On Mon, Sep 20, 2010 at 9:26 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>>> IMHO, most of this has been resolved by means of the XML-RPC
>>>> interface via the .release_data() method:
>>>>
>>>> http://wiki.python.org/moin/PyPiXmlRpc
>>>>
>>>> If there's something missing, we should add it there.
>>>
>>> Very clearly, most of egg-info is missing there. However, I
>>> sense that people are very opposed to adding it there.
>>>
>>> I merely asked whether it was ok to expose the egg-info
>>> data at all, and was met with strong opposition - nobody
>>> actually asked what specific API to expose them I had in
>>> mind.
>>
>> If the use case is to provide a way to build a dependency graph just
>> by querying PyPI, that's exactly what we have in mind in pep 345.
>
> I think I lost you there... .release_data() does provide the
> PEP 345 meta-data, at least according to the documentation.

Yes. I am the one who added it there.

And the information contained will let you have the list of
dependencies for a packages whatever your platform is -- that's the
goal of the work we did in the metadata area.

So uploading another metadata format like egg_info that contains
platform-dependent information to provide the same feature but in a
non-accurate way, is to be avoided -- no matter how it's exposed. As
other mentioned, it is often similar on every platform. but there's no
other mechanism than running egg_info again to be 100% sure of that..

True, there's no package yet that is using PEP 345 since its being
released now. But that's not a good reason imho to push new features
at PyPI based on the old format, knowing that these features are
existing in the new one, and knowing that the old format has issues as
explained.

Like py2 vs py3, I think all part of the packaging eco-system should
focus its energy on the new things we are building and just maintain
the old format and avoid spending energy and time on it. But that's my
own personal opinion :-)


Regards
Tarek
-- 
Tarek Ziad? | http://ziade.org

From monitor at jacobian.org  Mon Sep 20 22:58:57 2010
From: monitor at jacobian.org (monitor at jacobian.org)
Date: Mon, 20 Sep 2010 15:58:57 -0500
Subject: [Catalog-sig] [monit] pypi.python.org - Connection failed
Message-ID: <1285016342.1@jacobian.org>

Connection failed Service pypi.python.org 

	Date:        Mon, 20 Sep 2010 15:58:57 -0500
	Action:      alert
	Host:        jacobian.org
	Description: failed protocol test [HTTP] at INET[pypi.python.org:80] via TCP

Your faithful employee,
monit


From pje at telecommunity.com  Mon Sep 20 23:20:17 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 20 Sep 2010 17:20:17 -0400
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <4C97B567.5070104@v.loewis.de>
References: <4C93B6D8.1090700@v.loewis.de>
	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>
	<4C97AC5F.90005@egenix.com> <4C97B567.5070104@v.loewis.de>
Message-ID: <20100920212008.A3B233A40F5@sparrow.telecommunity.com>

At 09:26 PM 9/20/2010 +0200, Martin v. L?wis wrote:
>That's how I would have implemented it. Alas, this very
>idea was just shot down.

Hm.  When the issue was adding comments to PyPI, you mostly ignored 
the strong opposition from people who didn't want to use the feature, 
in order to support those who did.

While I'm only +0 (at most) on adding egg_info, I'm not sure why the 
objections of people who won't be using the feature should carry more 
weight in the present discussion than in the previous one.  ;-)


From martin at v.loewis.de  Mon Sep 20 23:49:36 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 20 Sep 2010 23:49:36 +0200
Subject: [Catalog-sig] egg_info in PyPI
In-Reply-To: <20100920212008.A3B233A40F5@sparrow.telecommunity.com>
References: <4C93B6D8.1090700@v.loewis.de>
	<AANLkTikgZDTCZSyeCOZJuZTc4=G444awLB6WjMEuLD+1@mail.gmail.com>
	<4C97AC5F.90005@egenix.com> <4C97B567.5070104@v.loewis.de>
	<20100920212008.A3B233A40F5@sparrow.telecommunity.com>
Message-ID: <4C97D6F0.70601@v.loewis.de>

Am 20.09.2010 23:20, schrieb P.J. Eby:
> At 09:26 PM 9/20/2010 +0200, Martin v. L?wis wrote:
>> That's how I would have implemented it. Alas, this very
>> idea was just shot down.
> 
> Hm.  When the issue was adding comments to PyPI, you mostly ignored the
> strong opposition from people who didn't want to use the feature, in
> order to support those who did.

Back then, I firmly believed that the feature was absolutely important
to have. Now, I admit that people can get the information if they
download all sdists. If there is demand for it, someone could also
provide a service that extracts all sdists, and provides the files
off-site, to reduce bandwidth usage for end-users.

So unlike with the comment system, the world is not going to end without
the feature.

> While I'm only +0 (at most) on adding egg_info, I'm not sure why the
> objections of people who won't be using the feature should carry more
> weight in the present discussion than in the previous one.  ;-)

If I understand correctly, it's more than that, though (and I would
easily continue ignoring people who object to a feature just because
they are not going to use it).

I feel that the core of the objection (from Tarek in particular) is
that it is viewed as directly competing to PEP 345. While I personally
think that this objection is flawed (i.e. exposing the data will not
have any significant impact on the PEP 345 adoption rate), I also
value Tarek's recent contributions to PyPI, and accept his doubts
(even though I don't share them).

Regards,
Martin


From merwok at netwok.org  Tue Sep 21 00:38:30 2010
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Tue, 21 Sep 2010 00:38:30 +0200
Subject: [Catalog-sig] PyPI mirror selection
In-Reply-To: <4C97B47A.7060409@v.loewis.de>
References: <4C9724B2.9060904@v.loewis.de> <4C9785DD.6030506@netwok.org>
	<4C97B47A.7060409@v.loewis.de>
Message-ID: <4C97E266.1090804@netwok.org>

>> Any plans to package the lib and publish it on PyPI?  (Maybe with a
>> more specific name.)
> Sure, will do (but I'd like to implement the missing features first).
Nice.

>> It seems very few edits would be needed to make the lib compatible with
>> 3.x.
I suppose you want to add missing features before being 3.x-compatible.

>> What is the minimum 2.x version you want to support?

Regards


From monitor at jacobian.org  Mon Sep 20 23:07:59 2010
From: monitor at jacobian.org (monitor at jacobian.org)
Date: Mon, 20 Sep 2010 16:07:59 -0500
Subject: [Catalog-sig] [monit] pypi.python.org - Connection succeeded
Message-ID: <1285016881.1@jacobian.org>

Connection succeeded Service pypi.python.org 

	Date:        Mon, 20 Sep 2010 16:07:59 -0500
	Action:      alert
	Host:        jacobian.org
	Description: connection succeeded to INET[pypi.python.org:80] via TCP

Your faithful employee,
monit


From swamidass at gmail.com  Thu Sep 23 20:43:47 2010
From: swamidass at gmail.com (S Joshua Swamidass)
Date: Thu, 23 Sep 2010 13:43:47 -0500
Subject: [Catalog-sig] trove classifiers
Message-ID: <AANLkTini7CUF-LJb8R8g+=f9UJ+jMmdZcsXzpsb_==L_@mail.gmail.com>

Would it be possible to create a new trove classifier for Scons
(http://www.scons.org) plugins?

Thanks!

S Joshua Swamidass
http://swami.wustl.edu/

From martin at v.loewis.de  Thu Sep 23 23:47:30 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 23 Sep 2010 23:47:30 +0200
Subject: [Catalog-sig] trove classifiers
In-Reply-To: <AANLkTini7CUF-LJb8R8g+=f9UJ+jMmdZcsXzpsb_==L_@mail.gmail.com>
References: <AANLkTini7CUF-LJb8R8g+=f9UJ+jMmdZcsXzpsb_==L_@mail.gmail.com>
Message-ID: <4C9BCAF2.8020806@v.loewis.de>

Am 23.09.2010 20:43, schrieb S Joshua Swamidass:
> Would it be possible to create a new trove classifier for Scons
> (http://www.scons.org) plugins?

What (specific) packages would be classified under that classifier?

Regards,
Martin