[Python-Dev] Packaging and binary distributions for Python 3.3

[Éric Araujo]

Le 09/10/2011 13:54, Paul Moore a écrit :
> On 9 October 2011 08:15, Éric Araujo <merwok at netwok.org> wrote:
>> Are there that many distributions with extension modules?  sdists should
>> work well even on Windows for pure Python projects.
> [...]
> Looking at my installations, I see:
> - database drivers (cx_Oracle, in my case)
> - lxml
> - pywin32
> - pyQT
> - pyzmq (that's just for playing a bit with IPython, so doesn't really count...)
> - I've also used in the past PIL mod_python (mod_wsgi more recently)
> and wxPython,

These are good examples.  Even if the number is not high, they are
widely used, so support for binary distributions really seems needed.

(When pip switches to distutils2 as underlying lib, they’ll get bdist
support for free!  IOW, proper bdist support would be another argument
to make the world switch to the new standards.)

> The pysetup features for uninstalling packages aren't going to work with
> bdist_wininst/bdist_msi (that's an assumption, I haven't tried them
> but I can't see how they would, and it'd certainly be a lot of
> marginally-useful effort to do even if it were possible).
> 
> The virtual environment stuff also wouldn't work that well with the
> installers, because they wouldn't have any way of finding which
> environments existed to ask where to install to. The same problem
> exists with virtualenv. (Again this is speculation backed by a small
> amount of playing with virtualenv, so I may be wrong here).

wininst and msi bdists can continue to be used as previously, for people
who want clicky installation to their system Python.  With built-in
package management and virtual environments in 3.3+, we can just
recommend that people publish bdist_simple instead of wininst or eggs.

> It may be that the bdist_dumb format would be OK. I haven't checked it
> out (to be honest, I don't think it's ever been used much).
It may or may not be.  bdist_dumb just makes a tarball or zipfile of
Python modules and built extension modules and is supposed to be
unpacked under sys.prefix or sys.exec_prefix.  However, that won’t play
nice with install options (--home, --user, --install-lib or redefined
--prefix) or sysconfig categories (i.e. I may want config files under
/usr/local/etc, scripts in /usr/local/bin, etc.).  I think we could
revamp bdist_dumb so that it’s really “sdist with compiled files” and
then we let pysetup install things to the right places.

>> Yes!  We need feedback to provide a much better tool than distutils,
>> before the API is locked by backward compatibility rules.
> Always the chicken and egg problem :-)
I’d rather say it’s “code in the stdlib has one foot in the grave” +
“stdlib code without active maintainer is effectively frozen” (hi
asyncore changes in 2.6!).

>> I actually wanted to talk about that, so let me take the opportunity.
>> What if we released packaging in Python 3.3 (and distutils2 1.0 on PyPI)
>> as a not-quite-final release?  [...]
> My immediate thought is that it would actually put people off using
> packaging for 3.3, they'd wait until "it is stable".
OK.  Too bad.

I’ll probably post that question again in its own message to get more
feedback.

> What is the status of distutils2? Is that (still?) intended to be
> effectively a backport of packaging to earlier Python versions?
Yes.  It works with 2.4-3.3.  I maintain it synchronized with packaging
in 3.3.  There are a small number of test failures which needs fixing
before I release distutils2 1.0a4 on PyPI.

> I'd suggest getting a distutils2 release available, and promoted,
> as the "early adopter" version of packaging.
We may do that, but I fear we’re going to lack time for that.  As part
of the stdlib, the packaging module API will be frozen in June, for the
first 3.3 beta.  We still have a lot to do: defining __all__ in all
public modules, changing some important internals (Tarek wants to kill
the subcommands system for example), fixing a number of bugs which may
imply incompatible API changes, etc.  The pace of development has slowed
much these last months, so I’m not sure we’ll reach 1.0 status months
before June.

> Maybe even with an option to install it as "packaging" so that people
> can use it in 3.2 and earlier and expect to need no changes when 3.3
> is released.
Not gonna happen!  d2 changed name on purpose when entering the stdlib,
so that in the future code can choose to use packaging (in the stdlib,
for example 1.0) or distutils2 (external backport, possibly of 2.0).
Code wanting to use “packaging if available otherwise distutils2” will
use the same import try/except as what’s done with unittest/unittest2,
json/simplejson and similar.

> A python-announce article "Python 3.3 new packaging features - early
> adopter release" publicising it, would be what I'm thinking of...
Here’s a plan:
1) make the docs usable (I’m on it)
2) fix the three test failures we currently have in d2
3) test on Windows and Mac
4) release 1.0a4 (I’ll do it)
5) announce and request feedback
6) work frantically before Python 3.3b1 to improve stuff and limit the
public API so as not to lock ourselves

Regards