[Distutils] Comments on PEP 426

[Daniel Holth]

On Fri, Sep 6, 2013 at 12:10 PM, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
> someone wrote:
>>  even though plain distutils would work just fine for a very
>> large subset of packages.  Most of my own packages
>> only use setuptools for its dependency resolution, and
>> that's for the most part provided by pip in the future.
>
>
> Another comment from the peanut gallery: This is very, very true -- I know
> nothing of distutils internal design or extensibility, but from the user
> end, it is really, really nice!
>
> It's actually quite remarkable that it "just works" most of the time -- even
> building universal binaries on OS-X, and needing very little platform
> dependent code in a setup.py.
>
> So what problems are we really trying to solve?
>
> For my part: I use setuptools for "develop" mode. That's it. Really.
>
> Then there is pip and easy_install -- nice to have the auto-discovery,
> download and dependency tracking -- though all of that is actually pretty
> over-rated -- it makes it faster to install a complex, dependency-riddled
> package (Like a web framework), but it breaks often enough that I end up
> doing it by hand anyway. For me, "pip install" really only works well for
> single, pure-python packages. (OK, some with external-dependency-free
> extensions, but only because I compile my own stuff so have set up the
> environment already -- and it's distutils that's working well there...)
>
> On the other hand, setuptools introduced a lot of cruft that seems to get in
> the way more than anything else.
>
> - Package versioning is a great idea and _should_ be a standard part of
> python, but the community seems to have settled on virtualenv as the
> solution to that. And the setuptools solution never seemed to work for me.
>
> - eggs: what's the point? and why mix a distribution format with an
> installation, and binary with source, and zipped with unzipped?
>
> - In general, setuptools seems to mix what should be install-time stuff with
> run-time stuff -- makes it very hard to use things like py2exe, py2app, etc.
> (pkg_resources is a mess, still not sure why it's needed)
>
> OK: so what are problems that really do need to be solved?
>
> Standard binaries that work with virtualenv -- as virtualenv seems to be the
> solution to versioning, we really need to be able to either download and
> install a binary in a virtualenv, or have pip do it for us -- wheels and the
> proper naming schemes seems to be going in the right direction for this.
>
> The big one: ALL of the problems I have with building compiled stuff is due
> to non-python external dependencies. Someone on the thread recently laid out
> all the ugly issues with different compilers, etc on OS-X, Windows, etc,
> etc. Where those all cause problems is the non-python dependencies: libpng,
> libfreetype, libnetcdf, libhdf, etc, etc, etc. That's where it gets ugly,
> and AFAIK, no one is trying to build a python solution (the only one I know
> of is gattai, which I've started using and hacking on, l but it really is
> just away to standardise your build scripts). Granted, I don't know that
> it's possible or desirable to have a python solution for that -- that's what
> apt-get, rpm, homebrew, macports, etc, are for, but that's where the real
> hang-ups are. Binary wheels _may_ help that a lot -- so very few folks
> actually need to deal with it.
>
> Sorry to sound critical here, but I'm hoping that with all this work, and
> failed startes (distutils2) we won't just end up in the same over-engineered
> crufty place we've been.
>
> All that said -- I'm looking forward to wheels and a pip that understands
> them -- IIUC, we have that, so it's on to the social part -- getting people
> to test/use them!
>
> I guess this reads like a rant, which is not really my intent -- it's great
> to see all the work getting done, and people thinking hard about good
> solutions. I really do appreciate all that effort.
>
> -Chris

distutils also doesn't have an effective (implemented) syntax for
declaring package dependencies inside setup.py.  If your package has
dependencies then it is most likely using setuptools.

distutils' compiler interface is probably less problematic than its
command architecture. David C has recently mentioned having to monkey
patch it just to change a compiler flag though.

In setuptools pkg_resources is used for runtime dependency resolution,
adding eggs to sys.path, entry_points plugin lookup, namespace package
support, finding a package's data files... it does a lot. It has also
made a number of people angry over the years.

pip, easy_install use a very dumb "wrong" greedy algorithm for
dependency resolution. We need SAT solvers like those used in PHP's
composer or Suse's libzypp that can take all the constraints into
account simultaneously and find an optimal solution but the problem of
composing many parts into a system will remain hard.

buildout does package versioning rather well without using virtualenv
and it scales to very complex projects.

A system called conda / anaconda goes as far as managing the Python
interpreter, the database, etc. required by what is described as a
C-level virtualenv. It may be a good choice if you do not want to be
stuck in the OS vendor's dll hell.

We are at least now mostly trying to break the work into bits. You are
likely to see proposals that solve a specific problem while leaving
others unsolved.

Daniel