[Distutils] formencode as .egg in Debian ??

[Phillip J. Eby]

At 09:54 PM 11/22/2005 +0100, M.-A. Lemburg wrote:
>In order to keep compatibilty with the existing wide-spread
>approach to install packages in site-packages/ using "python setup.py
>install", it should be possible (and I believe this should be the
>default to not disrupt existing usage and documentation) to
>run "python setup.py install" with an eggified source
>distribution in addition to the command to install it as
>regular egg.

"python setup.py install" *is* the command to install it as an egg - and 
the layout is no different than what occurs with an 'extra_path' setup() 
today.  The only difference of any consequence is that it maintains a 
*single* .pth file for all eggs, rather than one .pth file per egg, and 
that's an improvement by the standard you and Martin have put forth, where 
saving startup time is a good thing.


>I'm not suggesting to port over all the features you
>get from using setuptools' eggs (even though I do believe
>that you can go a long way using a special egg import
>hook),

Eggs don't need an import hook, which is one reason why they're so useful - 
they use the existing, well-established platform provided by Python 2.3 and 
above.


>  but it should be possible to get a regular working
>installation using "python setup.py install".

You already do!  It's just that *dependencies* also need to be installed as 
eggs, in order for the depending package to take advantage.  Without that, 
the depending package can't guarantee what version of the dependency is in 
use, thereby increasing the developer's support load.  (Kevin Dangoor, the 
TurboGears author, has previously mentioned that he couldn't imagine trying 
to support TurboGears without eggs, simply due to the overhead of debugging 
people's dependency problems on multiple platforms and packaging 
systems.  Even the support overhead caused by him making mistakes in using 
setuptools or by bugs in setuptools itself, was still dwarfed by the number 
of trouble-free installations on Windows and Mac OS.

And over the last few months, I believe we've also succeeded in stomping 
most of the issues that people had with getting solid non-root 
installations on their Linux distributions.  So the reasons for developers 
to prefer their dependencies to be managed as eggs will only improve over 
time, as the egg system allows Python developers to control and introspect 
their dependencies, rather than keeping that information hidden behind 
diverse platform-specific packaging tools.

So, the issue being surfaced here is that Python developers who want to use 
other people's code will want the other people's code packaged as eggs, 
even if their own package doesn't need to be an egg itself (other than to 
take advantage of the automatic dependency handling).  That's why 
easy_install exists: it can build eggs for most PyPI packages, even if the 
package author never heard of Python eggs.


>PS: I understand setuptools and eggs as feature set which
>adds functionality to distutils, not as competitive
>and disjoint all-in-one solution.

It's not disjoint at all, since it's trivial to package the majority of 
plain ol' distutils packages as eggs.  It isn't even competitive, in the 
sense that you are not forced to choose between one or the other.  You can 
have a "legacy" version of a package installed in the traditional way, and 
an egg version too.  Code that isn't using the dependency management 
facility will import the legacy version, and code that does will see the 
egg version.

It's only "competitive" if you feel that there must be only one way to do 
it.  (And if you do feel that way, then it also should be obvious that eggs 
are the superior solution, since they don't take away any capabilities of 
the old, only provide new ones.)


>The latter won't fly
>well with installations that require native installers
>to be used such as Debian's apt-get, rpm and all
>the others.

Currently, there is an experimental Gentoo "ebuild", and there's 
easy_deb.  I'm talking with some people about putting together an 
"easy_rpm" equivalent.  I recently heard that the easy_deb project was 
backed by Ubuntu, so it sounds like there's a fair amount of interest in 
supporting it out there.  Mainly, I see this as an issue of the packagers 
having to adapt to meet their users' changing needs.  It's reasonable to 
expect the major vendors to lag behind in terms of egg support, so there 
will naturally be a transitional period where tools like easy_deb and the 
hypothetical easy_rpm will be necessary.  (Setuptools' "bdist_rpm" command 
builds egg-based RPMs already, but a separate easy_rpm that can also 
rpm/eggify legacy packages would be less invasive.)

At the same time, it's also not reasonable for vendors to expect that 
ignoring eggs will make them go away - the practical advantages are far too 
compelling.  A developer who targets the egg runtime gets cross-platform 
dependency management and has the option of doing distributed development 
tracking subversion releases of their dependencies, and doesn't have to 
worry about the specifics of how those dependencies will get resolved.

He or she also gets to treat nearly all of PyPI as effectively his or her 
"standard library".  This is a major advantage over developers who do not 
do this, not only in developer effectivness, but also because a developer 
who depends exclusively on a specific packaging system will not have the 
same effective reach for their offering, or conversely will require a 
greater investment of effort to support various packaging systems.

To top it all off, the egg approach opens the possibility for system 
packagers to make use of developer-provided dependencies, thereby making 
the packaging process for Python packages potentially more robust.  The 
only packages for which additional dependency information is required are 
Python extensions that link to system libraries -- and even for those, I 
expect that we can find a packager-independent way to add a lot of that 
information to the egg metadata.

Obviously, every individual distribution would like to have Python packages 
conform to their individual system.  However, on the whole, it is clearly 
better for the Python developer to have practical dependency management 
that doesn't tie their efforts to a single platform, packaging system, or 
distribution.