Sharing Python installation between architectures

[Paul Smith]

On Fri, 2013-11-15 at 18:00 -0500, Paul Smith wrote:
> By this I mean, basically, multiple architectures (Linux, Solaris,
> MacOSX, even Windows) sharing the same $prefix/lib/python2.7 directory.
> The large majority of the contents there are completely portable across
> architectures (aren't they?) so why should I have to duplicate many
> megabytes worth of files?

OK, after some investigation and reading the code in Modules/getpath.c
to determine exactly how sys.prefix and sys.exec_prefix are computed (it
would be nice if this algorithm was documented somewhere... maybe it is
but I couldn't find it) I have a solution for this that appears to be
working fairly well.

In order to get this to work you need to use the following arguments
when you run "configure" to build your Python:

    configure --prefix=$PREFIX --exec-prefix=$PREFIX/$ARCH

where $ARCH can be pretty much whatever you want, but should be unique
for each different architecture of course.  The $PREFIX should be the
same for all architectures.

The magic here is ensuring that the --exec-prefix directory is a
SUBDIRECTORY of --prefix.  If that's not true, nothing works!

The resulting python interpreter will live in $PREFIX/$ARCH/bin: you
have to leave it there!  If you move it nothing works.  Although you can
get rid of the "bin" and move it up into $PREFIX/$ARCH if you want;
that's OK.

What I do is have a little shell-script wrapper installed somewhere else
that runs 'exec $EXECPREFIX/bin/python "$@"'

You can also correctly install extra packages with setup.py, even if
those packages have their own shared objects (like pycrypto or
whatever).

I should say, I've not thought about Windows yet.  I don't know if this
will work out for Windows.  However, Windows is such a different beast
anyway I think (at least in my environment) it will be OK to treat it
separately and require Windows people to download/install their own
Python.


There are few nitty things that I needed to handle:

     1. The _sysconfigdata.py file is put into $PREFIX not $EXECPREFIX,
        which is wrong since that file is very much
        architecture-specific.  As a post-processing step I moved it
        from $PREFIX/... into $EXECPREFIX/.../lib-dynload.  It's not
        quite correct since it's not a dynamic object, but lib-dynload
        is the only standard path on sys.paths from $EXECPREFIX.  It
        works OK anyway.
     2. All the scripts, even the ones in $PREFIX/bin, have hardcoded #!
        paths which go to a specific python in $EXECPREFIX/bin which is
        wrong (they can't be shared that way).  I use a simple "sed -i"
        to replace them all with "#!/usr/bin/env python" instead.
     3. There are some scripts that get dumped into $EXECPREFIX/bin
        rather than into $PREFIX/bin: "2to3", "idle", "pydoc",
        "smtpd.py".  I think this is simply a bug in the installation
        and those should all go into $PREFIX/bin.

Another weird thing is that the normal installation (this has nothing to
do with the above; it happens even if you don't set --exec-prefix)
contains TWO copies of libpython2.7.a; one in $EXECPREFIX/lib and one in
$EXECPREFIX/lib/python2.7/config.  These are over 14M each so it's not
inconsequential to have two.

I'm deleting the one in lib/python2.7/config and things still seem to
work OK.  The pkgconfig python definition references the one in
$EXECPREFIX/lib.