[Distutils] PEP439 and backward compat / easy_install / distlib

[Erik Bray]

On Tue, Mar 26, 2013 at 12:46 PM, Daniel Holth <dholth at gmail.com> wrote:
> On Tue, Mar 26, 2013 at 11:45 AM, Erik Bray <erik.m.bray at gmail.com> wrote:
>> On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth <dholth at gmail.com> wrote:
>>> My vision for the setuptools deprecation process is that distutils
>>> rides into the sunset with it. In this future eventually bugs in
>>> setuptools will be solved by porting setup.py to one of (X, Y, Z)
>>> which haven't necessarily been invented yet.
>>
>> That would be nice (really!) but what are you proposing replace it for
>> building packages with heavy reliance on C extensions?  Because for
>> that one use case (and perhaps that alone) it works "pretty okay" for
>> most cases.  I don't want to start seeing an infinite number of ways
>> to configure and build extension modules.  The great thing about using
>> distutils (or some variant) for this is that if I had the source for a
>> package I could just `./setup.py build` and it would "just work" for
>> all but the most complex cases (SciPy for example).
>>
>> I don't want to have a situation where some projects are using bento
>> and others are using scons and some are using waf and others are using
>> autoconf, etc, etc.  It's fine if a few projects have their own
>> special needs for build toolchains and I've been saying all along that
>> building should be separate from installing, and it should be easier
>> to drop in one's own build system.
>>
>> Another thing that setuptools provides that currently works "pretty
>> well" with extension modules is `./setup.py develop`.  It calls
>> `setup.py build_ext --inplace` to make extension modules importable.
>> Any build system for extension modules needs to be able to do
>> something similar to support in-place install functionality like
>> `setup.py develop`, `pip install -e`, etc.
>
> It's true that de-standardization of the build process has its own
> problems. As a consolation you don't have to do it as often because we
> have binary packages. It's hard to say how much fragmentation will
> happen, but distutils Extension() is an awful way to compile things!
> It has very little to recommend it apart from that it's there. For
> example, no parallel builds and no partial re-compiles based on what's
> changed. The trouble is that we know the packages on the cheeseshop
> mostly work but it's harder to count the stuff that avoids the
> cheeseshop because setuptools and setup.py wasn't a good solution.
>
> The example I've had to deal with recently is pycairo. They already
> use waf to compile their Python extension and don't use setup.py at
> all, so I argue that de-standardization has already happened. Whatever
> very easy (for pycairo) thing we can do make them "pip install"-able
> again is a plus.
>
> Even my own simple bcrypt wrapper "cryptacular" would compile an
> assembler file if I knew how to make that work with distutils. It
> doesn't and it's a little slower than it would have been if I had a
> decent build tool.
>
> I suspect at least 80% of packages will use some simple thing that
> comes with Python, two third party build tools will dominate, and we
> will discover interesting things that just weren't possible before. At
> least if someone wants to improve packaging we can make it easy for
> them to try without having to ask distutils-sig.

I pretty much agree with you on all of this, but I don't think the
question should be ignored either--avoiding this question is one of
the things that got previous packaging reform efforts into trouble.
Though the agreement to treat "build" and "installation" as two
different stories mitigates the issue this time around.  In any case
it's sort of off topic for this thread so I'll bring it up again
elsewhen.  One thing I see as a possible short-term solution is to
still rely on some version of distutils as a build tool *only*.  But
it would still be nice to have some easy way to standardize "in-place"
installation regardless of how extension modules get built.

Erik

P.S. pycairo does have a setup.py which worked for me, but the
installation instructions say it's "unsupported", though I don't see
the waf script doing anything enormously different from it.