[Python-Dev] issue 9807 - a glitch in coexisting builds of different types

[Barry Warsaw]

I am continuing my quest to be able to install multiple versions and builds of
Python simultaneously, so that they all nicely coexist.  E.g. you could have a
"normal" build of Python 3.2 and a --with-wide-unicode --with-pydebug build
both installed and all packages with extensions would get built and loaded for
the proper binary, etc.

(Caveat: I'm only intending to do the following work for configure-based
builds.)

I'm tracking most of this work in this bug:

    http://bugs.python.org/issue9807

I have a Bazaar branch from py3k which is pretty close to working, and I've
put my work so far up for review on Rietveld (see the bug comments for
details).

I've hit a snag though and I'm not sure what the best way out is.  I'm hoping
one of you have some better ideas.  First, let me set up the scenario.

Let's say I build py3k like this:

    % ./configure --prefix=/tmp/python && make altinstall
    % make distclean
    % ./configure --prefix=/tmp/python --with-wide-unicode --with-pydebug \
      && make altinstall

With my branch, you'll end up with this in /tmp/python:

    bin/python3.2m   - the normal build binary
    bin/python3.2dmu - the wide+pydebug build binary
    bin/python3.2m-config
    bin/python3.2dmu-config
    ...
    lib/libpython3.2.so.1.0.m
    lib/libpython3.2.so.1.0.dmum
    lib/python3.2m/config/Makefile
    lib/python3.2dmu/config/Makefile
    ...
    include/python3.2m/pyconfig.h
    include/python3.2dmu/pyconfig.h

and of course lots more.  This essentially segregates the build-specific bits
and pieces according to the $SOABI_QUALIFIER flags (build-flags) calculated in
the configure script.  This, along with the work for PEP 3149 gives us very
nice coexistence without too much duplication.  I.e.

    lib/python3.2/site-packages/foo-0.0-py3.2-plat.egg

can contain a shared .py file for all the builds, and .so files with PEP 3149
tags that distinguish between the various build flags.

Ah, but now here's the problem I'm stuck on.

Let's say I build the foo module, which has an _foo extension for both the
3.2m and 3.2dmu builds.  Everything gets installed correctly, and you'll see:

    lib/python3.2/site-packages/foo-...egg/_foo.cpython-32m.so
    lib/python3.2/site-packages/foo-...egg/_foo.cpython-32dmu.so

If you now run bin/python3.2dmu and try to import _foo, the right thing
happens (foreshadowing) assuming that the directory listing for the foo.egg is
alphabetical.

But what happens if you run bin/python3.2m and import _foo?  Yeah, well, even
though there's a _foo.cpython-32m.so sitting right there waiting for you to
load it, what actually happens is that Python tries to dynload
_foo.cpython-32dmu.so and crashes miserably.  Why did it do that?

The reason is that the import.c logic that uses the struct filedescr tables
built from _PyImport_DynLoadFiletab are just not smart enough to handle this
case.  All it knows about are suffix, and for backwards compatibility, we have
dynload_shlib.c matching both .SOABI.so *and* bare .so.  So when it's
searching the directories for .cpython-32m.so files, it finds the
.cpython-32dmu.so first based on the bare .so match.

I can think of a couple of ways out, none of which are totally satisfying.
Probably the easiest out is to change the PEP 3149 naming so that the files
don't end in ".so".  E.g. use this instead:

    foo.cpython-32dmu-so
    foo.cpython-32m-so

or similar.  I think that'd be a fairly straightforward change, but it might
break some useful assumptions (we'd still fall back to .so of course).

Other ideas:

- make import.c smarter so that you can match against other than just the
  suffix.  probably a lot of work.

- extend struct filedescr to add callbacks into dynload_shlib.c for file
  matching.  this callback would reject .SOABI.so files where the SOABI
  doesn't match.  again a lot of work and more expensive due to tests more
  complicated than simple strcmps.

- scan filesystem for .SOABI.so before .so.  you lose locality of reference
  and probably has lots of other really nasty semantic disturbances. ;)

- remove match on bare .so.  not good for non-distutils based build
  processes.

Do you have any other ideas?  If not, what would your preference from the
above be?  Right now I think I'd go with changing the PEP 3149 naming scheme
to not end in .so (i.e. option #1).

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20101001/bfbca55b/attachment.pgp>