[Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata

[P.J. Eby]

At 07:15 PM 7/6/2009 +0100, Paul Moore wrote:
>My point was that distributions which use setuptools in their
>setup.py, *even if there's no runtime dependency on setuptools* end up
>with non-standard .egg-info's. There is no good reason for this, from
>my POV as a package user. So if setuptools is breaking otherwise
>entirely standard packages, so that they won't work with core
>distutils mechanisms, that's wrong. Setuptools is refusing to play
>nicely with other packaging systems.
>
>Now at the moment, this is a moot point, because the stdlib doesn't
>have any package discovery mechanisms.

Not so; it has .egg-info.  (See footnote below.)


>The PEP 302 hooks I'm writing provide the "plugin" ability for
>alternative storage formats. But there's no provision for alternative
>*metadata* formats. What's the point of a standard if people don't
>follow it?

I'm confused.  Are you saying pkgutil will recognize .egg-info, or not?

The only practical difference between the existing formats and the 
proposed new one (AFAICR) is the extension of the directory name, and 
the presence or absence of the RECORD file.

My POV is that it would be a shame to discard compatibility with 
existing installed software, just for a bikeshedding of the directory 
name.  Especially if it means that any tools that currently generate 
.egg-info's (and I'm not 100% positive setuptools is the only case) 
will need to grow options for whether to generate the old name, 
possibly based on what Python version...  except you can install new 
distutils into old Python... etc.

If this were strictly a matter of, "from Python 3.x on, generate 
.pydist", it's a lot less of a problem for me and everybody else, 
because Python 3 is effectively going to force a fork (or at least a 
branch) of setuptools anyway.

For Python 2.x, having *two* names for .egg-info (which is what I 
understand the proposal to effectively be) is just asking for trouble, IMO.

And my concern is that currently these compatibility issues are being 
handwaved rather than considered up front.

To reiterate, summarize, and attempt to make as concrete as I can:

I'm fine with a rename, *if and only if* it can be done in a way that 
never requires a tool to generate BOTH files for compatibility 
reasons.  AFAICT, this can only be done by making the extension 
dependent upon the Python version, and this version distinction must 
be made *even if* the distutils themselves are backported to an 
earlier Python version.  (Alternatively, this could be resolved by 
*not* backporting the PEP 376-compatible distutils/pkgutil to 2.x.)

Currently, there is no official Python 3 release of setuptools, so 
compatibility can (IMO) be considered moot there.  What's more, it is 
easy to conditionally determine whether to use '.egg-info' or 
'.pydist' based on Python version.  What is not easy and likely to be 
confusing for everyone, is having the possibility for *both* to exist 
in the same system, *without* also having bidirectional interoperability.

So, I'm +1 for no change (obviously), and +0 for "always .pydist in 
3.x and always .egg-info in 2.x", whether the latter part is achieved 
by making distutils/pkgutil use a version-dependent extension, or by 
refusing to backport distutils/pkgutil to 2.x.  I'm -1 for having 
.pydist ever appear on a 2.x Python, either via stdlib or backporting.

(These comments also apply for any name other than '.pydist'.)

Footnote: there has been some discussion that implies that somehow 
.egg-info was hijacked into the stdlib, and that it should therefore 
not be supported.  This is incorrect: at Pycon the year this happened 
(2005 IIRC), Guido *blessed setuptools for inclusion in the stdlib* 
in his keynote.  Yes, *all* of setuptools, including easy_install, 
eggs, the whole nine yards.

A few months later, when I actually checked in the code, there were 
protests from a few people on Python-Dev, some of which I agreed were 
valid.  I reverted the checkin, floating instead a compromise 
proposal to add .egg-info support to distutils (i.e. the 
install_egg_info command) to allow setuptools and other 3rd party 
tools to detect the presence of distutils-installed packages.

At the time, the assumption was that eggs and setuptools would still, 
eventually, make it into the stdlib -- at the time, Guido had not yet 
changed his mind about that.  So the ".egg-info" name was not 
controversial, to the best of my recollection, and the inclusion did 
not violate any of the expressed concerns of the three persons who 
objected to including setuptools whole-hog.

So, from my POV at least, dropping .egg-info is dropping backward 
compatibility on a feature that was *intentionally* added to the 
stdlib -- with approval from Guido and non-protest from Python-Dev -- 
specifically to provide that compatibility.

That doesn't obligate it to stay...  but I suggest that it *does* 
mean Python 3, not 2, is the appropriate place for dropping it (if it 
is indeed to be dropped)...  and that it is not appropriate to do so 
in 2.x on merely a handwave by persons not involved in the previous 
discussions.  (And of course Guido can Pronounce otherwise; I'm just 
saying I don't think anything less than such a pronouncement is 
required to remove it from 2.x.  And again, I'm fine with making a 
clean break of it in 3.x, since setuptools is already broken there anyway.)