python 2.7.x on MacOSX: failed dlopen() on .so's

[Paul Smith]

On Wed, 2013-11-13 at 23:06 -0800, Ned Deily wrote:
> On Nov 13, 2013, at 17:24 , Paul Smith <paul at mad-scientist.net> wrote:
> > I'm discovering that this is tricky.  I don't want to bring OS wars into
> > it, but this kind of thing is so simple and just works on GNU/Linux.  I
> > guess I've been spoiled :-).
> 
> Well, if you are building from scratch, you would likely run into some
> of the same issues on Linux systems, too.  There are many variations
> in configurations that are mutually incompatible.

Possibly.  However as long as you're using basic system libraries and
not higher-level stuff like GTK etc., you can compile on an older
version of Linux (say, Red Hat EL 5 or so) and the result will run
without any problems on pretty much any distribution, even the newest
ones: Red Hat, Ubuntu, Debian, Suse, etc. etc.  We do this regularly,
for production code we ship to customers.  About the only "basic"
libraries that cause problems are libssl/libcrypto... the openssl folks
just don't seem to care much about this issue.

> > I'm somewhat dreading my next effort after MacOS: the same thing, on
> > Windows :-/.
> > 
> > And another task, which seems like it will be fun: building GDB on
> > MacOSX with Python support enabled...
> 
> Keep in mind that on newer OS X releases, Apple no longer ships gcc or
> gdb.  The standard compiler is clang and it has its own debuggers.

Yes, I'm aware.  Very frustrating.  We have a large set of specialized
macros, both in native GDB and in Python using GDB's extension, that are
invaluable for debugging our code.  I'm not thrilled about the idea of
trying to get all that working in lldb as well.  So we're going to stick
with GDB, building it ourselves, until it no longer works at all.  At
that point we'll decide how to proceed.

> > Hm, that's an idea.  I don't HAVE to build Python myself, actually, I
> > just need (a) it to be relocatable, and (b) to add these extra modules
> > to it so I can use it across systems without installing them
> > individually by hand.
> 
> I'm not sure I know what you mean by "relocatable".  Like the
> third-party package manager solutions, the python.org installers place
> the Pythons they install in fixed locations (under /Library/Frameworks
> with links from /usr/local/bin) but one that does not conflict with
> any system files, including the Apple-supplied system Pythons.

By relocatable I mean "runnable from any location"; i.e., not fixed.  I
have a wrapper around the Python executable that can compute the correct
root directory and set any environment variables or add flags or
whatever might be needed.

Basically I have a large number of test systems and a set of test suites
that are all written in Python, and I need to be able to distribute the
same version of Python plus a specific set of additional modules across
all those systems to be sure they all have the same environment.  Plus
the set of systems changes (new systems added/old ones removed)
regularly.  I'm handling this by checking in the Python distribution
plus modules into a "tools" Git repository, then cloning it on each
system.  However, I have no special privileges on these systems (so I
can't modify any system locations such as /usr/local) and I can't
control what user account will be running the tests (so the user's home
directory, where the tools repository is cloned, is not constant across
the systems).

I have this working on Linux with very little effort.  Now I'm trying to
get the same result on MacOSX.