[Python-Dev] Packaging and binary distributions for Python 3.3

[Paul Moore]

On 14 October 2011 15:13, "Martin v. Löwis" <martin at v.loewis.de> wrote:
>> Thanks for the clarification. I can see why this would be important.
>> But maintaining 3 different interfaces to do essentially the same
>> thing (collect some data from the user, then based on that data put
>> the same set of files in the same places) seems a waste of effort, and
>> a recipe for discrepancies in capabilities.
>>
>> Maybe the wininst and MSI installers should ultimately become simple
>> UIs around a zipfile and an invocation of the packaging APIs? Not that
>> I'm offering to do that work, I'm afraid...
>
> I think you are mixing issues: even if they were simple wrappers, they
> would still be 3 different interfaces (presented to the user). I.e.
> the user doesn't really care whether there is a zip file inside
> (as in bdist_wininst) or a cab file (as in bdist_msi), and whether or
> not the packaging APIs are invoked during installation.
>
> So if you want to get rid of interfaces, you really have to drop
> one of the formats.
>
> Making maintenance of the interfaces simpler and more homogenous
> by having them call into installed Python code is certainly worthwhile.
> The challenge here is that the installers also work on older Python
> versions (unless there are extension modules in there), so they
> could only use the packaging API when installing into 3.3 or newer.

You are right that I'm mixing issues somewhat. I think the two issues
are multiple interfaces and multiple formats.

- On interfaces, I personally don't mind the existence of multiple
choices. Some people like a GUI, others like command lines. Some like
platform integration, others like integration with the language
environment, or a consistent cross-platform experience. Trying to
mandate one interface will always upset someone. So I don't see this
as an important goal in itself. (But see below for a proviso [1]).

- On formats, I strongly believe that having multiple formats is a
problem. But I need to be clear here - an installer (MSI, wininst) is
a bundle containing executable code (which drives the interface), plus
a chunk of data that is the objects to be installed. (I am
oversimplifying here, but bear with me). That's necessary because the
package must be a single download, but it means that the delivery
format combines code and data. Format to me refers to that data (and
how accessible it is).

The problem is that the packager has two roles:
1. He does the build, which is an essential service as end users don't
necessarily have the means to compile.
2. He chooses the distribution format (and hence the user experience),
which is not essential per se.

The second role is the problem, as the packager's preferences may not
match those of the end user.

I am proposing decoupling those 2 roles. In the purest sense, I'd like
to see a distribution format that contains *purely* the binaries, with
no UI intelligence. That's what the proposed bdist_simple format is
about. This format can be consumed by external tools that just read it
as data (that's what pysetup, and packaging, does).

For a standard user experience on Windows, at least, there needs to be
a means of wrapping that data into an installer. MSI is the obvious
choice, because it's the MS standard, but there's a problem here in
that there's no obvious way to retrieve the raw bdist_simple file from
the MSI once you've bundled it. The wininst format is better here, as
you can retrieve the original format (just by removing, or ignoring,
the EXE header).

I have no opinions on how the MSI/wininst installers should present
options to the user, nor do I have any strong views on how they should
do the installation (my instinct says that they should reuse the
packaging core code, but as you say that causes compatibility
problems, so may not be feasible).

My only strong views are:
- *if* there are multiple formats, they should be easily convertible
- pysetup should be able to consume one of the formats (trivially if a
bdist_simple format exists, less so if MSI becomes the only format...)

The second point is because pysetup offers capabilities that
MSI/wininst do not - at least right now, and probably always, as I
expect the core packaging code to support new features like venvs
sooner than Windows-specfic formats. If only because Unix developers
can hack on packaging.

Conversion between bdist_wininst and the proposed bdist_simple format
is easy as they are both fundamentally zipfiles. The MSI format is the
odd one out here, as I don't know how to build an MSI from binary data
(without using bdist_msi - or to put it another way, the layout of the
files that bdist_msi expects to build the installer from isn't
documented) and I don't know how to get binaries out of an MSI
(without installing). If these two things were possible, we'd have the
means to convert between formats at will (and writing a tool that
non-packagers could use to convert formats would be possible).

I'm focusing on writing code to allow packaging to install from a
binary zipfile for a very simple reason - it's easy! The code is in
Python, most of the support routines are there, and it's just a matter
of putting the bits together. I had a mostly-working prototype done in
a couple of hours last night. In contrast, I wouldn't even know where
to start to modify bdist_wininst or bdist_msi to conform to the new
package metadata standards, let alone to cope with non-system Pythons
(which pysetup does out of the box). So by developing a new format, I
can at least offer code to help, rather than just flooding the mailing
lists with suggestions (although I seem to be doing that as well :-))

Paul.

[1] The one sticking point is that users generally want all the
features that exist, so if bdist_msi doesn't support installing into
"non-system" Python builds, and pysetup does, that will drive people
to either switch, or more likely complain about the "limitations" of
MSI :-)