[Distutils] [Fwd: Re: Annoucing distribute project]

[Ian Bicking]

Phillip J. Eby wrote:
> In the case of the distutils, the people who are capable of extending it 
> are tired of doing so, and the people who have the energy and time are 
> very unlikely to be able to work on it much without breaking something 
> significant.  Distutils is also far too flexible in some areas to be 
> able to improve much while maintaining 100% backward compatibility -- it 
> doesn't enforce One Obvious Way To Do It.

Basically volunteer projects like this just won't move forward when 
doing so is painful -- I think distutils is like this, and though you 
suffered through that with the development of setuptools I think part of 
this recent discussion is that it's still not really self-sustaining. 
So I think we need a bit higher bar of code quality (and brevity) than 
in a commercial project.

> What's more, there are very few people who've even said they like the 
> distutils API or think it's a good fit for the application area.  And, 
> frankly, the domain knowledge embedded in the distutils are of fairly 
> limited scope and kind:
> 
> * Extension building, compile/link options and defines
> * Wildly-differing installation path schemes across platforms
> * Platform distribution formats like bdist_rpm, bdist_wininst, and 
> bdist_msi
> 
> Most other things the distutils do are either well-specified, obsolete 
> (e.g. its internal logging and option parsing libs), or probably not 
> worth keeping (e.g. bdist_dumb).

Yes -- what distutils really does just isn't that interesting or 
complicated.  There's a lot of arcane knowledge built into the build 
process, but it's not well expressed or particularly useful, and for 
pure-Python packages there really isn't any arcane knowledge to be had. 
  There's other features, but a lot of them aren't well enough 
implemented to be reliable, or they are adapted in eclectic ways by 
individual packages in a way that makes them unpredictable.  The arcane 
knowledge is distutils is broken and unusable -- I don't think distutils 
reform would actually save much that was useful.

>> > Maybe a "distutils 2" project could start outside Python, using 
>> distutils
>> > and setuptools code as
>> > legacy infrastructure, and taking back pieces of it,
>> >
>> > Then it could be re-integrated in Python as a replacement for 
>> distutils when
>> > it is mature ?
>>
>> Only if much effort and study went into the planning of this 
>> re-integration.
> 
> That's why at least some of the discussion has been around requirements 
> gathering and PEP-writing as a first step.

There is a much better sense of what the scope of distutils should be 
now than when it was written, and the PEP process could make that 
clarity explicit.  I suspect that when the scope is made clear that the 
implementation just won't be that complex.

> My own inclination is that a scalable future for distutils means an 
> improved sdist format, the end of setup.py as an command-line interface, 
> and community-maintained platform-specific installation tools that 
> process source or binary distributions.  Most complaints about distutils 
> (and setuptools, for that matter) are focused on installation 
> policy&preference issues.  Making it possible and practical for a 
> variety of tools to flourish around a standardized format (ala WSGI) 
> seems like the way to go.

I'd like to see an improvement of sdist, toward a more declarative and 
introspectable system.  Just including EGG-INFO in sdist would be a big 
help, and allow the easy reading of the egg metadata without having to 
deal with the binary aspects of .egg (which remain very challenging to 
use reliably).  In pyinstall I'm unpacking and running egg_info to get 
this data, but that's very awkward and that metadata should really just 
be there without having to invoke any code.

I'm not sure I agree with removing setup.py, though I dislike how it 
functions currently -- that you kind of have to poke it from the side to 
get basic information out of it, instead of something more declarative. 
  In some ways it's the distutils/setuptools interface that most people 
actually use -- pyinstall for instance only calls setup.py in 
subprocesses to install the package, and never touches it directly.  So 
anything that acts fairly similar to the current setup.py's will be 
compatible.  This is part of why I don't think the backward 
compatibility issue is as difficult as Guido suggests.

Though... really if pyinstall could easily detect a new source format 
and only setup.py with the old source format, it wouldn't be hard to 
support both.  There does need to be room for custom code, but mostly 
for the build/compilation process.

> Notice that the existence of eggs has already allowed buildout, 
> virtualenv, and pyinstall to appear.  But eggs don't handle installing 
> tests or documentation, and they have to be prebuilt by platform.  An 
> improved sdist format, on the other hand, with standardized layout and 
> metadata would address all of those issues.
> 
> The tools for building this format and APIs for inspecting it would be 
> candidates for the stdlib, in much the same way that wsgiref was - the 
> spec is/should be stable, and those parts that are 
> compiler/install-layout specific will need to be maintained in the 
> stdlib anyway, for Python's own build infrastructure.
> 
> In that sense, "distutils 2" would not be so much a rewrite of the 
> distutils, as the separation of them into tools for distributing, and 
> tools for installing, where some of the tools for installing may be 
> community-maintained.

I agree.  For instance, I don't see a particular advantage to making 
pyinstall part of the core of Python -- it would increase adoption and 
add a certain review process, but its usefulness is not contingent on 
everyone using it.  It's more important that there's a consensus (or... 
movement towards some consensus) about how people write and distribute 
packages.  If some distutils 2 effort just addressed that, and avoided 
things like installation that mostly can be avoided (though installation 
needs to be co-developed, of course), it should keep the scope down so 
that it's not quite as hard to agree on things.

-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org