[Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)

[Paul Moore]

2009/10/8 Tarek Ziadé <ziade.tarek at gmail.com>:
> Here's a quick summary of the main things that are going to happen in
> Distutils, and Distribute, and a few words on virtualenv and pip.
> (there is much much more work going on, but I don't want to drown
> people with details)

Thanks for this summary. The overview was getting lost in all the details.

> == What's installed ? what's the installation format ? how to uninstall ? ==
>
> First, it's an incomplete package manager : you can install a
> distribution using it, but there's no way to list installed
> distributions. Worst, you can't uninstall a distribution.
>
> PEP 376 resolves this, and once it's finished, the goal is to include
> the APIs described there into Distutils itself and into the pkgutil
> module in stdlib. Notice that there's an implementation at
> http://bitbucket.org/tarek/pep376 that is kept up to date with PEP 376
> so people can see what we are talking about.
>
> Another problem that popped during the last years is the fact that, in
> the same site-packages, depending on the tool that was used to install
> a Distribution, and depending if this distribution uses Distutils or
> Setuptools, you can have different installation formats.
>
> End-users end up with zipped eggs (one file), unzipped eggs (one
> self-contained format in a directory) and regular Distutils (packages
> and modules in site-packages). And the Metadata are also located in
> many different places depending on the installation format used.
>
> That can't be. there's no point to keep various installation format in
> the *same* site-packages directory.
>
> PEP 376 also resolves this by describing a *unique* format that works
> for all. Once this is finished, Distutils will implement it by
> changing the install command accordingly.
>
> - Work left to do in PEP 376 : restrict its scope to a disk-based,
> file-based site-packages.
> - Goal: 2.7 / 3.2

+1

This addresses my biggest concern with the way PEP 376 was going
(largely prompted by my suggestions, so I'll take the blame for this!)
which was that PEP 376 was trying to define list/uninstall methods for
package formats (i.e. zip files/eggs) that it couldn't install itself.

I presume that when you refer to the restriction in scope, you do mean
that PEP 375 will lose all support for zip files, as well as the
generalised PEP 302 support I proposed? I still believe that
special-casing zip files is wrong - PEP 302 deliberately abstracted
the import mechanisms and built zipfile support on top of that, and I
still believe that's a good thing. But expanding PEP 302 to cover the
requirements of PEP 376 looked more and more inappropriate the more I
wrote code to do it - what would be needed would be a *new* protocol
on top of which zip-based formats could be layered. I'm quite happy
for that to be outside the scope of PEP 376, though, as distutils
itself doesn't support any such formats.

The egg format

One thing missing from your roadmap (unless I missed it) is the fate
of the egg (zipfile) format. If it's to remain a valid option
(bdist_egg and the like) then I'd assume that Distribute needs to be
the place to develop it. One thing it will need is an equivalent to
PEP 376 introspection and uninstall features. And *that* is where I
believe a standard protocol belongs - developed based on the real use
requirements for egg files, with a suitable file-based layer which can
be added into distutils (somewhat like importlib does for PEP 302)
once the protocol is agreed and standardised.

Paul.