[Distutils] [Fwd: Re: Annoucing distribute project]

[Phillip J. Eby]

At 03:24 PM 9/25/2008 -0700, Guido van Rossum wrote:
> > Right now there's a momentum in the community, including framework gurus,
> > that are
> > willing to work on a new distutils package. They are not core 
> developers but
> > they are really
> > good in distribution matters. Even Phillip Eby said that starting a new
> > distutils could be a good
> > pick in this thread earlier.
>
>I wasn't there. I'd like to refer to a post by Joel Spolsky about the
>problem with total rewrites:
>http://www.joelonsoftware.com/articles/fog0000000069.html

The economic factors are a bit different, here.  Joel himself has 
previously pointed out that where Netscape failed, Mozilla won - 
i.e., the economics of open source can mean that it's sometimes 
easier to get volunteers for a new project than for fixing an old 
one, or at least for a project where dropping backward compatibility 
is allowed (e.g. Py3K).

In the case of the distutils, the people who are capable of extending 
it are tired of doing so, and the people who have the energy and time 
are very unlikely to be able to work on it much without breaking 
something significant.  Distutils is also far too flexible in some 
areas to be able to improve much while maintaining 100% backward 
compatibility -- it doesn't enforce One Obvious Way To Do It.

What's more, there are very few people who've even said they like the 
distutils API or think it's a good fit for the application 
area.  And, frankly, the domain knowledge embedded in the distutils 
are of fairly limited scope and kind:

* Extension building, compile/link options and defines
* Wildly-differing installation path schemes across platforms
* Platform distribution formats like bdist_rpm, bdist_wininst, and bdist_msi

Most other things the distutils do are either well-specified, 
obsolete (e.g. its internal logging and option parsing libs), or 
probably not worth keeping (e.g. bdist_dumb).


> > Maybe a "distutils 2" project could start outside Python, using distutils
> > and setuptools code as
> > legacy infrastructure, and taking back pieces of it,
> >
> > Then it could be re-integrated in Python as a replacement for 
> distutils when
> > it is mature ?
>
>Only if much effort and study went into the planning of this re-integration.

That's why at least some of the discussion has been around 
requirements gathering and PEP-writing as a first step.

My own inclination is that a scalable future for distutils means an 
improved sdist format, the end of setup.py as an command-line 
interface, and community-maintained platform-specific installation 
tools that process source or binary distributions.  Most complaints 
about distutils (and setuptools, for that matter) are focused on 
installation policy&preference issues.  Making it possible and 
practical for a variety of tools to flourish around a standardized 
format (ala WSGI) seems like the way to go.

Notice that the existence of eggs has already allowed buildout, 
virtualenv, and pyinstall to appear.  But eggs don't handle 
installing tests or documentation, and they have to be prebuilt by 
platform.  An improved sdist format, on the other hand, with 
standardized layout and metadata would address all of those issues.

The tools for building this format and APIs for inspecting it would 
be candidates for the stdlib, in much the same way that wsgiref was - 
the spec is/should be stable, and those parts that are 
compiler/install-layout specific will need to be maintained in the 
stdlib anyway, for Python's own build infrastructure.

In that sense, "distutils 2" would not be so much a rewrite of the 
distutils, as the separation of them into tools for distributing, and 
tools for installing, where some of the tools for installing may be 
community-maintained.

That's the general idea, anyway.

(I'm going to be away from email for a few days, so I'll probably be 
out of this thread 'till Tuesday.)