[Python-Dev] Status of packaging in 3.3

[Nick Coghlan]

On Thu, Jun 21, 2012 at 3:29 AM, PJ Eby <pje at telecommunity.com> wrote:
> On Wed, Jun 20, 2012 at 9:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On Wed, Jun 20, 2012 at 9:46 PM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>> > Agreed, especially if the "proven in the wild" criterion is required
>> > (people won't rush to another third-party distutils replacement, IMHO).
>>
>> The existence of setuptools means that "proven in the wild" is never
>> going to fly - a whole lot of people use setuptools and easy_install
>> happily, because they just don't care about the downsides it has in
>> terms of loss of control of a system configuration.
>
>
> Um, this may be a smidge off topic, but what "loss of control" are we
> talking about here?  AFAIK, there isn't anything it does that you can't
> override with command line options or the config file.  (In most cases,
> standard distutils options or config files.)  Do you just mean that most
> people use the defaults and don't care about there being other options?  And
> if that's the case, which other options are you referring to?

No, I mean there are design choices in setuptools that explain why
many people don't like it and are irritated when software they want to
use depends on it without a good reason. Clearly articulating the
reasons that "just include setuptools" is no longer being considered
as an option should be one of the goals of any PEPs associated with
adding packaging back for 3.4.

The reasons I'm personally aware of:
- it's a unilateral runtime fork of the standard library that bears a
lot of responsibility for the ongoing feature freeze in distutils.
Standard assumptions about the behaviour of site and distutils cease
to be valid once setuptools is installed
- overuse of "*.pth" files and the associated sys.path changes for all
Python programs running on a system. setuptools gleefully encourages
the inclusion of non-trivial code snippets in *.pth files that will be
executed by all programs.
- advocacy for the "egg" format and the associated sys.path changes
that result for all Python programs running on a system
- too much magic that is enabled by default and is hard to switch off
(e.g. http://rhodesmill.org/brandon/2009/eby-magic/)

System administrators (and developers that think like system
administrators when it comes to configuration management) *hate* what
setuptools (and setuptools based installers) can do to their systems.
It doesn't matter that package developers don't *have* to do those
things - what matters is that the needs and concerns of system
administrators simply don't appear to have been anywhere on the radar
when setuptools was being designed. (If those concerns actually were
taken into account at some point, it's sure hard to tell from the end
result and the choices of default behaviour)

setuptools is a masterful achievement built on shaky foundations that
will work most of the time. However, when it doesn't work, you're
probably screwed, and as soon as it's present on a system, you know
that your assumptions about understanding the Python interpreter's
startup sequences are probably off. The efforts around
distutils2/packaging have been focused on taking the time to *fix the
foundations first* rather than accepting the inevitable shortcomings
of trying to build something in the middle of a swamp.

> If the long-term goal is to draw setuptools users over to packaging, then
> AFAIK the packaging effort is still missing a few things, like build-time
> dependencies and alternatives to setuptools' entry points and "extras", as
> well as the ability to integrate version control for building sdists
> (without requiring the sdist's recipient to *also* have the version control
> integration in order to build the package or recreate a new sdist).

Right - clearly enumerating the features that draw people to use
setuptools over just using distutils should be a key element in any
PEP for 3.4

I honestly think a big part of why packaging ended up being incomplete
for 3.3 is that we still don't have a clearly documented answer to two
critical questions:
1. Why do people choose setuptools over distutils?
2. What's wrong with setuptools that meant the idea of including it
directly in the stdlib was ultimately dropped and eventually replaced
with the goal of incorporating distutils2?

I imagine there are answers to both of those questions embedded in
past python-dev, distutils-sig, setuptools and distutils2 mailing list
discussions, but that's no substitute for having them clearly
documented in a PEP (or PEPs, given the scope of the questions).

We've tried to shortcircuit this process twice now, first with "just
include setuptools" back around 2.5, and again now with "just include
distutils2 as packaging" for 3.3. It hasn't worked, so maybe it's time
to try doing it properly and clearly articulating the desired end
result. If the end goal is "the bulk of the setuptools feature set
without the problematic features and default behaviours that make
system administrators break out the torches and pitchforks", then we
should *write that down* (and spell out the implications) rather than
assuming that everyone knows the purpose of the exercise.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia