[Python-Dev] Packaging and binary distributions

[Paul Moore]

I'd like to reopen the discussions on how the new packaging module
will handle/support binary distributions in Python 3.3. The previous
thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html)
included a lot of good information and discussion, but ultimately
didn't reach any firm conclusions.

First question - is this a Windows only problem, or do Unix/MacOS
users want binary support? My feeling is that it's not an issue for
them, at least not enough that anyone has done anything about it in
the past, so I'll focus on Windows here.

Second question - is there a problem at all? For the majority of
Windows users, I suspect not. The existing bdist_wininst and bdist_msi
formats have worked fine for a long time, offer Windows integration
and a GUI installer, and in the case of MSI offer options for
integrating with corporate distribution policies that some users
consider significant, if not essential. (Binary eggs are a third, and
somewhat odd, case - a number of projects have started distributing
binary eggs, but I don't know what benefits they have over
bdist_wininst in particular, as easy_install will read bdist_wininst
installers. Perhaps a setuptools/distribute user could comment. For
now I'll assume that binary eggs will slowly go away as packaging gets
more widely adopted).

So that leaves a minority who (1) prefer integration with packaging,
(2) need to work with virtual environments or custom local builds, (3)
need binary extensions in some or all of their environments and (4)
don't want to have to build all the binaries they need from scratch.

Given the scale of the issue, it seems likely that putting significant
effort into addressing it is unwise. In particular, it seems unlikely
that developers are going to move en masse to a new distribution
format just to cater for this minority. On the other hand, for people
who care, the fact that packaging (currently) offers no direct support
for consuming binary distributions is a fairly obvious hole. And
having to build from source just to install into a virtual environment
could be a showstopper.

The bdist_wininst format is relatively amenable to manipulation - it's
little more than a zip file, after all. So writing 3rd party code to
install the contents via packaging shouldn't be hard (I've done some
proof of concept work, and it isn't :-)) Vinay's proposal to use the
resource mechanism and some custom hooks would work, but I'd like to
see a small amount of extra direct support added to packaging to make
things cleaner. Also, if packaging supported plugins to recognise new
distribution formats, this would make it possible to integrate the
extra code seamlessly.

The MSI format is a little more tricky, mainly because it is a more
complex format and (as far as I can tell from a brief check) files are
stored in the opaque CAB format, so the only way of getting data out
is to do a temporary install somewhere. But I see no reason why that
isn't achievable.

So, my proposal is as follows:

1. I will write a 3rd party module to take bsist_wininst and bdist_msi
modules and install them using packaging
2. Where packaging changes are useful to make installing binaries
easier, I'll request them (by supplying patches)
3. I'll look at creating a format-handling plugin mechanism for
packaging. If it's viable, I'll post patches
4. If it seems useful, my module could be integrated into the core
packaging module

I don't intend to do anything about a GUI, or modify the existing
formats at all. These don't interest me, particularly, so I'll leave
them to someone who has a clear picture of what they want in those
areas, and the time to develop it.

For 3.3 at least, I'd expect developers to continue distributing
bdist_wininst or bdist_msi format files. We'll see what happens with
binary eggs.

Unix/MacOS users who care will need to propose something themselves.

Does anyone have any comments?

Paul.