[Distutils] Improving distutils vs redesigning it (was people want CPAN)

[David Cournapeau]

On Thu, Nov 12, 2009 at 5:01 PM, Lennart Regebro <regebro at gmail.com> wrote:

>
> What I'm confused about is that you say that distutils should be
> scrapped and not incrementally improved, yet your effort of doing it
> "the right way" builds on distutils.

Maybe we do not mean the same by rewriting a distribution tool then.
Numscons works by totally bypassing distutils and using its own
mechanism for anything build-related. It used to use compiler options
from distutils, but I have added a mode to bypass this entirely so
that you can use new compilers by writing scons tools without having
to care about distutils (e.g. to build numpy/scipy with Intel
compilers on mac os x or windows).

Now, I am trying to see how far I can go from a purely static file for
package description, plus tools to convert existing setup.py to it.
The hope is that I can soon get a tool which does build and packaging
without using distutils at all.

>
>> That's one of the reason why I find the claim of distutils
>> having a deep and large knowledge hard to believe.
>
> Me too, but I think the community does. And currently that knowledge
> is being focused into a bunch of PEP's. Which is the reason for my
> previous (and so far ignored) question if that work should be ignored,
> or if a new solution done The Right Way should build on those PEPs.

Here are the PEP I see being discussed related to anything
distribute/setuptools-related. Feel free to indicate me the ones I am
missing.

PEP 382: namespace packages

Mostly independent from the distribution tool, I have not followed the
discussion on those ones. The current setuptools implementation causes
deployment issues because of file sharing which is why I mention it
here.

PEP 345, 390, etc...: Metadata

I think a new format ala Cabal is needed for full package description.
The current format is ok for distribute evolution, but retrofitting a
full description feels very hackish. The one I am working on in
toydist is such as many packages only require this file, no setup at
all (the format is purely straw-man, to see what kind of problems need
to be solved). Right now, I have enough code so that I can almost
convert several simple (but not trivial, e.g. I consider sphinx to be
a simple but not trivial package)  packages from pypi. This is done
through a distutils command, i.e. if you do:

python setup.py gen_toyinfo

with a setup like

"""
setup(name='foo',
 version='1.0',
 packages=['foo'],
 extensions=[Extension(_foo, sources=['bar.c', 'foo.c'])]
)
"""

you get a toysetup.info description file:

"""
Name: foo
Version: 1.0

PackageAdditionalData:
  data/*.dat
  test/*.py

Library:
  Extension: _foo
    src: bar.c, foo.c
  Packages: foo
"""

I have also code to build from this file alone using distutils and
scons at will: this file can be fed back to a trivial setup.py, so
here I get backward (and even forward if needed) compatibility. One
major problem is package data: the current way to use MANIFEST and co
is too complicated (and fragile when MANIFEST are not updated
correctly). I don't want to retrofit it because I strongly believe
distribution files should be explicitly added, but I guess it would
make sense to support it at conversion time only.

PEP 376: egg-info

I don't personally care about the egg and egg-info format details,
only that it is specified (and versioned), like ruby gems, for
interoperability. So that I can build eggs from scons, for example.

Enthought has some nice open source tools which give you reliable
install/upgrade/removal/listing of software using eggs as a format
(ala dpkg and apt-get, using eggs as the basic format). One of the
major problem with eggs is that on windows at least, some people
prefer using .exe/.msi because of its integration with add/remove. If
the egg format is rich enough, it should be possible to get tools to
convert from one format to the other.

> And  if it should, what the difference then is to the current effort
> of fixing distutils.
>
> If scons is the Right Way then it seems to me yet again that the whole
> discussion is moot, and that what should be done is to build
> extensions for scons to build/install/upload Python modules, and then
> after doing that, just saying "Use scons instead".

Clearly, requiring distutils, distribute or whatever to reimplement
scons or waf is insane. But today, numscons only works because it uses
many implementation details of distutils, and cannot be used as it is
because all the distribution/installing/packaging part from distutils
is not reusable. If distribute could be improved so that it could make
this easy, it would be great. But I believe doing so without being
disruptive is hard - and the only way to understand why is to try
doing it.

David