[Python-Dev] Packaging and binary distributions

[Paul Moore]

On 30 October 2011 23:17, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Paul Moore <p.f.moore <at> gmail.com> writes:
>
>> The MSI format is a little more tricky, mainly because it is a more
>> complex format and (as far as I can tell from a brief check) files are
>> stored in the opaque CAB format, so the only way of getting data out
>> is to do a temporary install somewhere. But I see no reason why that
>> isn't achievable.
>
> It's not just about getting the data out of the CAB, though - it's also about
> integration with Add/Remove Programs and the rest of the Windows Installer
> ecosystem.

Hang on. I'm talking here about repackaging the binary files in the
MSI file for use in a pysetup install invocation. As pysetup has no
GUI, and doesn't integrate with Add/Remove, there's no issue here. If
you want a GUI and Add/Remove integration, just run the MSI. Or am I
missing something? We seem to be at cross purposes here, I suspect I'm
missing your point.

>> 1. I will write a 3rd party module to take bsist_wininst and bdist_msi
>> modules and install them using packaging
>
> It would be important to retain the flexibility currently offered by setup.cfg
> hooks, as I don't believe any out-of-the-box approach will work for the wide
> range of use cases on Windows (think Powershell scripts, Visio templates and
> other Microsoft Office integration components).

Why? Again, if this is purely as a means to consume bdist_xxx files,
then the only flexibility needed is enough to cater for any variations
in data stored in the bdist_xxx format. The wininst format is easy
here - it has directories PLATLIB, PURELIB, DATA, SCRIPTS and HEADERS
(corresponding to the installation --install-xxx parameters) and
that's all. As long as the module is flexible enough to deal with
that, it can read anything bdist_wininst can produce.

> I'm also not sure if these formats provide all the flexibility required - e.g.
> they may be fine for extension modules, but how do they handle packaging include
> files?

Ah, I think I see what you are getting at. If someone uses the new
features and flexibility of packaging to create a fancy custom install
scheme, how do they bundle up a binary distribution from that? My
(current) answer is that I don't know. The packaging module as it
stands only offers the legacy bdist_xxx formats, so the answer is "run
pysetup run bdist_wininst on it". If that breaks (as it is likely to -
wininst format isn't very flexible) then tough, you're out of luck.

I 100% agree that having a "native" packaging means of building binary
distributions from source ones, which captures all of the necessary
information to cover any flexibility available to setup.cfg, would be
good. But that's potentially a much bigger project than I can manage.
My bdist_simple format was based off bdist_dumb/bdist_wininst and had
the same limitations as that.

You might be able to get somewhere by running build, then zipping up
the whole directory, source, build subdirectory and all. Then on the
target machine, unzip and do a --skip-build install. That's a bit of a
hack, but should in theory work. Whether it's the basis of a sensible
distribution format I don't know.

>> For 3.3 at least, I'd expect developers to continue distributing
>> bdist_wininst or bdist_msi format files. We'll see what happens with
>> binary eggs.
>>
>> Unix/MacOS users who care will need to propose something themselves.
>
> I'm not sure there's anything especially Windows-specific about the
> bdist_wininst format, apart from the prepended GUI executable. One drawback of
> any current scheme is that if you're packaging an extension module that runs on
> say Windows, Linux and Mac OS X, there's no easy way to build or distribute a
> single archive (for a given version of Python, say) which has all the binary
> variants you want to include, such that at installation time, only the bits
> relevant to the target platform are installed. The current packaging
> functionality does sort of support this, but it entails potentially tedious
> manual editing of the setup.cfg file to add information about what resources
> apply to which platform - the kind of tedious editing which would be obviated by
> the right kind of additional support code.

Again, I agree that this would be useful. Not something I have the
time to look at though (although if someone else picks it up, I'd be
interested in doing some testing and maybe contributing to the work).

I think I now see why we're not understanding each other. I'm coming
from the position that the projects I care about (as an end user) use
bdist_wininst or bdist_msi at the moment, so all I want is a way of
using, as a consumer, those existing distributions (or something
equivalent in power) to install the packages via pysetup (which gets
me the ability to install in development builds and venvs). I see why
a more powerful binary format would be nice for developers, but as an
end user I have no direct need for it.

Thanks for your patience.
Paul.