[Python-Dev] versioned .so files for Python 3.2

[Barry Warsaw]

I'm trying to catch up on this thread, so I may collapse some responses or
refer to points others have brought up.

On Jun 24, 2010, at 05:53 PM, Scott Dial wrote:

>If the package has .so files that aren't compatible with other version
>of python, then what is the motivation for placing that in a shared
>location (since it can't actually be shared)?

I think Matthias has described the motivation for the Debian/Ubuntu case, and
James describes Python's current search algorithm for a packages .py[c] and
.so files.  There are a few points that you've made that I want to respond to.

You claim that versioned .so files scheme is "more complicated" than multiple
version-specific search paths (if I understand your counter proposal
correctly).  It all depends on your point of view.  From mine, a 100 line
patch that almost nobody but (some) distros will care about or be affected by,
and that only changes a fairly obscure build-time configuration, is much
simpler than trying to make version-specific search paths work.

If you build Python from source, you do not care about this patch and you'll
never see its effects.  If you get Python on a distribution that only gives
you one version of Python at a time, you also will probably never care or see
the effects of this patch.  If you're a Debian or Ubuntu user who wants to use
Python 3.2 and 3.3, you *might* care about it, but most likely it'll just work
behind the scenes.  If you're a Python packager or work on the Python
infrastructure for one of those platforms, then you will care.

About just sharing the py files.  You say that would be acceptable to you, but
it's actually a pretty big deal.  If you're supporting two versions of Python,
then every distro Python package doubles in size.  Even with compression,
you're talking longer download times and probably more critically, you've
greatly increased CDROM space pressures.  The Ubuntu CDROM is already
essentially at capacity so doubling the size of all Python packages (most of
which btw do not have extension modules) makes such an approach impossible.
Moving to a DVD image has been discussed, but it is currently believed not in
the best interest of users, especially on slow links, to do so at this time.

The versioned .so approach will of course increase the size of packages by
twice the contained .so file size, and that's already an uncomfortable but
acceptable increase.  It's acceptable because of the gain users get by having
multiple versions of Python available and the fact that there aren't nearly as
many extension modules as there are Python files.  Doubling the size of .py
files as well isn't acceptable.

>But the only motivation for doing this with .pyc files is that the .py
>files are able to be shared, since the .pyc is an on-demand-generated,
>version-specific artifact (and not the source). The .so file is created
>offline by another toolchain, is version-specific, and presumably you
>are not suggesting that Python generate it on-demand.

Definitely not.  pyc files are generated upon installation of the distro
package, but of course the .so files must be compiled on a build machine and
included in the distro package.  The whole process is much simpler if the
versioned .so files can just live in the same directory.

>For packages that have .so files, won't the distro already have to build
>multiple copies of that package for all version of Python? So, why can't
>it place them in separate directories that are version-specific at that
>time? This is not the same as placing .py files that are
>version-agnostic into a version-agnostic location.

It's not a matter of "could", it's a matter of simplicity, and I think
versioned .so files are the simplest solution given all the constraints.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100630/275b8632/attachment-0001.pgp>