[Python-Dev] Status of packaging in 3.3

[Paul Moore]

On 20 June 2012 10:12, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I think the whole idea that distutils should be frozen and improvements
> should only go in distutils2 has been misled. Had distutils been
> improved instead, many of those enhancements would already have been
> available in 3.2 (and others would soon be released in 3.3).

The problem seems to be that in the past, any changes in distutils
have been met with cries of "you can't do that", basically because the
lack of a clearly distinguished extension API means that the
assumption is that for any bit of the internals, someone somewhere has
monkeypatched or relied on it. The issue is compounded by the fact
that a lot of distutils is undocumented, or at least badly documented,
so saying "if it's not documented, it's internal" doesn't work :-(

Maybe if we could be a bit more aggressive in saying what counts as
"internal" and resist the cries of "but I use it", modifying distutils
might be a more viable option, But there doesn't seem to be anyone
willing to take and defend that step. IIRC, Tarek proposed
distutils2/packaging after getting frustrated with how little he could
usefully do on distutils itself.

> Deciding to remove packaging from 3.3 is another instance of the same
> mistake, IMO.

I see your point, but without sufficient developer resource, the
question is whether packaging is in a usable state at all. Nobody but
Éric is actually working on packaging (and as he says, even he is not
able to at the moment), so what alternatives are there? I guess one
extra option not mentioned by Éric is to make the packaging issues
into release blocker bugs. That would effectively stall the release of
3.3 until packaging could be brought to an acceptable state,
effectively a form of blackmail. I can't imagine anyone wants to take
that approach. And yet, some of the existing bugs would clearly be
release blockers if they were in any other part of Python.

I think the first question is, do we need an enhanced distutils in the
stdlib? As far as I can see, this one has been answered strongly, in
the affirmative, a few times now. And yet, the need seems to be a
diffuse thing, with no real champion (Tarek and Éric have both tried
to take that role, and both appear to have been unable to invest the
necessary amount of time - which doesn't surprise me, it seems to be a
massive task).

Removing packaging from 3.3, in my mind acknowledges that as it stands
the approach was a failed experiment[1]. Better to get it taken out
before it appears in a released version of Python. We need to rethink
the approach. I see a number of options going forward, all of which
are based round the need to ensure enough developer involvement, so
that Tarek and Éric get help, and don't simply burn out before we have
anything useful.

1. Reconsider the decision to freeze distutils, with a view to
migrating incrementally to the feature set we want from packaging.
That'll be hard as we'd need to take a much stronger line on making
changes that could break existing code (stronger in the sense of "so
we broke your code, tough - you were using undocumented/internal
APIs"). And I suspect Tarek wouldn't be willing to take this route, so
we immediately lose one resource. Maybe the other core developers
could take up the slack, though. For example, Antoine, you seem to be
implying that you would have worked on distutils if this had happened.

2. Free up distutils2 to develop as an external package, and then have
a PEP proposing its inclusion in the stdlib in due course, when it is
ready and has been proven in the wild. The benefit here is that I
assume that as a separate project, becoming a committer would be
easier than becoming a Python core developer, so there's a wider pool
of developers. The downside is that the timescales would be a lot
longer (I doubt we'd see anything in 3.4 this way, and maybe not even
3.5).

3. Write a PEP describing precisely what the packaging module will do,
get consensus/agreement, and then restart development based on a solid
scope and spec. This is the correct route for getting something added
direct to the stdlib, and it's a shame it didn't happen in the first
place for packaging. Having said that, the PEP would likely be huge,
given the scope of packaging, and would require a lot of time from a
champion. There's no guarantee that championing a PEP wouldn't burn
someone out just as rapidly as developing the module itself :-( And
also, given that the packaging documentation is one of its weak areas,
I'd have to say I have concerns as to whether a PEP would come
together in any case... The assumption here, though, is that the PEP
process creates the debate, and results in interested parties coming
together in the discussion. If we can keep that momentum, we get a
pool of interested developers who may well assist in the coding
aspects.

The one option I don't like is taking packaging out, releasing 3.3,
and then putting it straight back in as is, and simply carry on as
now, in the hope that it'll be ready for 3.4. I honestly doubt that
the only issue is that we've run out of time before 3.3. There are
more fundamental problems that need to be addressed as well -
specifically the reliance on one individual to bear all of the load.

Just my thoughts,
Paul.

[1] That reads really harshly. I don't mean to criticise any of the
work that's been done, I'm a huge fan of the idea of packaging, and
its goals. The "experiment" in this case is around process -
developing something as big and important as packaging with limited
developer resource, relatively directly in the core (bringing it in
from distutils2 sooner rather than later) and working from a series of
smaller PEPs focused on particular details, rather than an overall one
covering the whole package.