[Distutils] RFC: Binary Distribution Format for distutils2/packaging

[Paul Moore]

On 14 March 2012 02:50, Carl Meyer <carl at oddbird.net> wrote:
> In terms of distribution format, this sounds great; I think a clean
> break and new format that existing systems will just naturally ignore is
> the way to go.

Agreed. I created a patch a while ago for a "bdist_simple" format,
which has languished in the tracker for a while. It's a very basic
patch, and the format is essentially just a stripped down version of
the wininst format (a zip file with distinguished placeholder
directories at the top level). I'm not sure I understand Jim's issues
with the wininst format, so I can't respond as to why this wouldn't
address them.

I'd started writing a PEP for bdist_simple with Éric Araujo but
stalled because I didn't have time. If someone else takes up the baton
with a binary distribution format, that's fine with me.

> Your message doesn't address the on-disk installed format. I hope that
> the installed format (at least the d2/p default) will be consistent with
> PEP 376 (unzipped and "flat"), not an egg-style importable zip file
> relying on its own dedicated entry in a pth file.

I agree that a flat on-disk format is essential. It's what pip and the
standard tools use. The .pth based egg format appears to only have
benefits if you want to manage multiple versions, and as far as I can
see that requirement is now better satisfied via virtualenv/pip.

One other point - it's actually relatively easy to write a tool to
introspect egg and wininst format files, and unpack them manually into
the on-disk layout needed. I have the bones of such a thing for pip
(needs completing before I can release it).

I would suggest that for backward compatibility and transition
purposes, the following things need to be considered:

- There should be converters from egg and wininst formats to the new
format. Not everyone will start producing the new format immediately,
and we should offer users a migration path where packagers haven't
moved over. These should be trivial to write.
- There should also be a converter from MSI format, but that's a beast
to write and I'd say not worth the effort.
- External tools like pip will need to handle the new format, ideally.
Or at least keep out of the way when pysetup install is used. I'm
assuming that pysetup install works in a virtualenv.

Is the intention to promote the new format for older Python versions
as well, via distutils2, or will this be a purely Python 3.3 option?
If the former, conversion and compatibility is crucial, if the latter,
then less so but we need to avoid causing problems for early adopter
users who rely on packagers who haven't moved yet ("Lobby the package
maintainer to change" is never a helpful answer...)

The key here is that the new format needs to be a unifying effort - we
already have 3 binary formats, if this new format merely adds a
fourth, then in my view it's a failure. It needs to be the "one
obvious way" if it's to work.

Paul.

PS A couple of relevant python-dev threads, for people who haven't seen them:

http://mail.python.org/pipermail/python-dev/2011-October/114285.html
http://mail.python.org/pipermail/python-dev/2011-October/113956.html
http://mail.python.org/pipermail/python-dev/2011-October/114243.html