[Python-Dev] Packaging and binary distributions for Python 3.3

[Paul Moore]

On 9 October 2011 08:15, Éric Araujo <merwok at netwok.org> wrote:
>> But one thing struck me - the "Installing Python Projects" document
>> talks about source distributions, but not much about binary
>> distributions.
> This is inherited from distutils docs, not a deliberate choice.  We just
> haven’t thought much, if at all, about binary distributions.
>
>> On Windows, binary distributions are significantly more important than
>> on Unix, because not all users have easy access to a compiler, and
>> more importantly, C library dependencies can be difficult to build,
>> hard to set up, and generally a pain to deal with.
> Are there that many distributions with extension modules?  sdists should
> work well even on Windows for pure Python projects.

To be honest, I'm not that sure. I tend to find that many of the ones
I want to use have binary dependencies, but maybe I'm atypical :-)

Looking at my installations, I see:
- database drivers (cx_Oracle, in my case)
- lxml
- pywin32
- pyQT
- pyzmq (that's just for playing a bit with IPython, so doesn't really count...)
- I've also used in the past PIL mod_python (mod_wsgi more recently)
and wxPython, but don't these days because they either aren't
available for Python 3, or no binaries are available and building them
is a pain.

I've hit others in the past, but mainly just in idle hacking, so I
don't depend on them as such (and can't really remember which).

>> I don't know what format packaging favours.
> As a direct distutils descendant, can create bdist_wininst and
> bdist_msi.  For installing, I was not aware of the problem you reported
> (“does not interact well with pysetup”); can you give more info?  I’m
> guessing it boils down to the fact that Windows binary installers are
> meant to be clicked by users, not managed with command-line tools.

Precisely that (and nothing really more). The pysetup features for
uninstalling packages aren't going to work with
bdist_wininst/bdist_msi (that's an assumption, I haven't tried them
but I can't see how they would, and it'd certainly be a lot of
marginally-useful effort to do even if it were possible).

The virtual environment stuff also wouldn't work that well with the
installers, because they wouldn't have any way of finding which
environments existed to ask where to install to. The same problem
exists with virtualenv. (Again this is speculation backed by a small
amount of playing with virtualenv, so I may be wrong here).


> IIRC the current behavior in pysetup is to favor source distributions,
> but bdists should probably be favored for non-pure distributions on Windows.
>
>> So there will be a need for a pysetup-friendly binary format.
>> I assume that the egg format will fill this role - or is that not the
>> case? What is the current thinking on binary distribution formats for
>> Python 3.3?
> First, we don’t want to include wholesale support for setuptools eggs in
> packaging.  We have options to support egg/egg-info metadata in the PEP
> 376 implementation (packaging.database), because we need that to provide
> a useful tool for users and help them switch, but eggs are another
> matter.  After all, if setuptools and then pkg_resources were turned
> down for inclusion in Python 2.5, it’s not now that we have packaging
> that we’ll change our mind and just bless eggs.  What we can do however
> is to see what bdist_egg does and define a new bdist command inspired by
> it, but without zipping, pkg_resource calls, etc.

It may be that the bdist_dumb format would be OK. I haven't checked it
out (to be honest, I don't think it's ever been used much). I could
have a play with that and see if it did the job (or could be made to).
Like you say, eggs have a lot of extra infrastructure that wouldn't be
needed here.

>> The main reason I am asking is that I would like to write an article
>> (or maybe a series of articles) for Python Insider, introducing the
>> new packaging facilities from the point of view of an end user with
>> straightforward needs (whether a package user just looking to manage a
>> set of installed packages, or a module author who just wants to
>> publish his code in a form that satisfies as many people as possible).
> That’s excellent.  I too thought about writing something about packaging
> for that blog, but an outside end-user viewpoint like yours would best
> match the readership.  I can write a shorter piece just for packaging
> tool developers (i.e. how to use packaging as a library), or you can
> write that one too and act as a tester for our doc and API.

Let's see how things go. My goal is to evangelise packaging so that
people writing packages I use will create binary builds to save me the
effort (:-)) - but anything else I can do, if I have the time, I'm
happy to chip in with.

>> What I'd hope to do is, as well as showing people all the nice things
>> they can expect to see in Python 3.3, to also start package authors
>> thinking about what they need to do to support their users under the
>> new system.
> Yes!  We need feedback to provide a much better tool than distutils,
> before the API is locked by backward compatibility rules.

Always the chicken and egg problem :-)

> I actually wanted to talk about that, so let me take the opportunity.
> What if we released packaging in Python 3.3 (and distutils2 1.0 on PyPI)
> as a not-quite-final release?  (Something like Python 3.0, which was not
> considered a real version and not supported as much as the other ones.)
>  The goal would be to expose it to a large range of users to get bug
> reports and feature requests, but without locking us forever into one
> API or design, which was the death of distutils a year ago.  The idea is
> not to scare people with warnings that we’ll break APIs on a whim, but
> that we keep the option to change parts of packaging and release a 2.0
> with Python 3.4, with documented changes from 3.3.  Opinions?

My immediate thought is that it would actually put people off using
packaging for 3.3, they'd wait until "it is stable".

What is the status of distutils2? Is that (still?) intended to be
effectively a backport of packaging to earlier Python versions? If so,
then I'd suggest getting a distutils2 release available, and promoted,
as the "early adopter" version of packaging. Maybe even with an option
to install it as "packaging" so that people can use it in 3.2 and
earlier and expect to need no changes when 3.3 is released. That would
have the usual "nobody bothers testing stuff that hasn't been released
yet" problems, but it might at least get some take-up. But maybe
that's what distutils2 already is, and it just needs more promotion?

A python-announce article "Python 3.3 new packaging features - early
adopter release" publicising it, would be what I'm thinking of...

Paul.