[Python-Dev] magic in setuptools (Was: setuptools in the stdlib)

[Phillip J. Eby]

At 10:59 PM 4/20/2006 +0200, Martin v. Löwis wrote:
>Bob Ippolito wrote:
> > They DO NOT compete any more than source packages do.  eggs are packages
> > plus metadata, nothing more.  What eggs do and what rpm/msi/deb does are
> > orthogonal.  It's entirely reasonable that in the future rpm/msi/deb
> > will simply be a delivery mechanism for eggs.
>
>That might be your view, but it apparently isn't the view of the
>inventor(s).

For the record, Bob *is* the co-inventor of eggs; he was the only person 
who responded to my call on the distutils-sig for the creation of a format 
for distribution of plugins, and it is largely due to his efforts that the 
egg runtime exists in the first place.  In addition, it was he who 
convinced me that they would be good for installing arbitrary Python 
libraries, not just plugins for extensible applications.

It's true that a lot of our initial discussions and even some of the 
documentation seem to blur the line a bit as to what eggs are.  Clarity 
often takes time to develop.  In fact, it wasn't until the Debian packaging 
flap on distutils-sig last fall that we were forced to clarify our thoughts 
enough to realize that distribution packaging was always just a means to an 
end - a way of reifying the idea of a library or plugin as a self-contained 
and self-describing unit.  Distribution and installation are certainly a 
part of the overall picture, but the essential nature of eggs is to make 
*project releases* truly first-class objects in Python.

A lot of your arguments (and some of MAL's) against eggs as "unnecessary" 
have been based on the (incorrect) theory that Python packages (importable 
unit) constitute the appropriate grain of modularity for system packages 
(distributable unit).  But ever since the distutils existed, it has been 
clear that distribution units are not a 1:1 mapping with packages in the 
Python sense of the word.  The stdlib itself gives lie to that concept, 
since there are many packages in it, yet it is distributed as a single 
project release.

Eggs, however represent project releases as tangible objects: the "1.1 
release of FooBar" is a thing that can be installed, put on sys.path, 
searched for, depended on, and so forth.  This is a valuable concept to 
have access to from Python code, independent of how these project releases 
actually end up on a system, or how they are physically arranged when they 
get there.

It is intimately related to delivery and installation, of course.  But to 
claim that they're competing with system packages is to miss the point.

If you run "bdist_rpm" on a setuptools-based setup script, for example, you 
will get an RPM that looks just like any other RPM for a Python package, 
merely with the addition of an .egg-info directory that contains 
introspectable data.  (And the same thing happens for bdist_wininst and any 
other bdist_* tool that uses "setup.py install --root" to build its target 
image.)

In other words, if the delivery and installation part is handled by some 
other tool, then only the true "egg" remains: a project release as a 
discoverable and introspectable unit.

That's not to say that the eggs didn't *originate* as a way to solve all of 
the problems in one fell swoop, but the point is that eggs do *NOT* compete 
with existing system packaging tools within their domain of competence.

What I'm opposed to in making setuptools install things the distutils way 
by default is that there is no easy path to clean upgrade or installation 
in the absence of a system packaging tool like RPM or deb or 
what-have-you.  I am not opposed to doing the "classic" style of 
installation by default *forever*, but only until setuptools can handle 
uninstallation and upgrades in that format.  Right now, it's a lot easier 
to uninstall things if they are all in one directory.