[Python-Dev] versioned .so files for Python 3.2

[Barry Warsaw]

This is a follow up to PEP 3147.  That PEP, already implemented in Python 3.2,
allows for Python source files from different Python versions to live together
in the same directory.  It does this by putting a magic tag in the .pyc file
name and placing the .pyc file in a __pycache__ directory.

Distros such as Debian and Ubuntu will use this to greatly simplifying
deploying Python, and Python applications and libraries.  Debian and Ubuntu
usually ship more than one version of Python, and currently have to play
complex games with symlinks to make this work.  PEP 3147 will go a long way to
eliminating the need for extra directories and symlinks.

One more thing I've found we need though, is a way to handled shared libraries
for extension modules.  Just as we can get name collisions on foo.pyc, we can
get collisions on foo.so.  We obviously cannot install foo.so built for Python
3.2 and foo.so built for Python 3.3 in the same location.  So symlink
nightmare's mini-me is back.

I have a fairly simple fix for this.  I'd actually be surprised if this hasn't
been discussed before, but teh Googles hasn't turned up anything.

The idea is to put the Python version number in the shared library file name,
and extend .so lookup to find these extended file names.  So for example, we'd
see foo.3.2.so instead, and Python would know how to dynload both that and the
traditional foo.so file too (for backward compatibility).

(On file naming: the original patch used foo.so.3.2 and that works just as
well, but I thought there might be tools that expect exactly a '.so' suffix,
so I changed it to put the Major.Minor version number to the left of the
extension.  The exact naming scheme is of course open to debate.)

This is a much simpler patch than PEP 3147, though I'm not 100% sure it's the
right approach.  The way this works is by modifying the configure and
Makefile.pre.in to put the version number in the $SO make variable.  Python
parses its (generated) Makefile to find $SO and it uses this deep in the
bowels of distutils to decide what suffix to use when writing shared libraries
built by 'python setup.py build_ext'.

This means the patched Python only writes versioned .so files by default.  I
personally don't see that as a problem, and it does not affect the test suite,
with the exception of one easily tweaked test.  I don't know if third party
tools will care.  The fact that traditional foo.so shared libraries will still
satisfy the import should be enough, I think.

The patch is currently Linux only, since I need this for Debian and Ubuntu and
wanted to keep the change narrow.

Other possible approaches:
 * Extend the distutils API so that the .so file extension can be passed in,
   instead of being essentially hardcoded to what Python's Makefile contains.
 * Keep the dynload_shlib.c change, but modify the Debian/Ubuntu build
   environment to pass in $SO to make (though the configure.in warning and
   sleep is a little annoying).
 * Add a ./configure option to enable this, which Debuntu's build would use.

The patch is available here:

    http://pastebin.ubuntu.com/454512/

and my working branch is here:

    https://code.edge.launchpad.net/~barry/python/sovers

Please let me know what you think.  I'm happy to just commit this to the py3k
branch if there are no objections <wink>.  I don't think a new PEP is in
order, but an update to PEP 3147 might make sense.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100624/cd9bde42/attachment.pgp>