python 2.7.x on MacOSX: failed dlopen() on .so's

[Ned Deily]

On Nov 13, 2013, at 17:24 , Paul Smith <paul at mad-scientist.net> wrote:

> On Wed, 2013-11-13 at 16:00 -0800, Ned Deily wrote:
>>> The reason I've set PYTHONHOME is ultimately I need this installation to
>>> be relocatable.  It's going to be shared across lots of different
>>> systems and they'll have the ability to copy it wherever they want.
>> 
>> That could be problematic. You need to be *really* careful about how
>> you do that.  You stand a chance with a non-shared installation.
> 
> You mean, --disable-shared?  That's what I want, ultimately, but I was
> going to start with the default configuration first.

--disable-shared is the default, the same as not specifying --enable-shared.

> I'm discovering that this is tricky.  I don't want to bring OS wars into
> it, but this kind of thing is so simple and just works on GNU/Linux.  I
> guess I've been spoiled :-).

Well, if you are building from scratch, you would likely run into some of the same issues on Linux systems, too.  There are many variations in configurations that are mutually incompatible.  If you mean "due to the lack of a system package manager" like yum or apt-get, there are various third-party package managers for OS X: MacPorts, homebrew, and fink being the most well-known.  They all provide Pythons and many third-party Python packages.

> I'm somewhat dreading my next effort after MacOS: the same thing, on
> Windows :-/.
> 
> And another task, which seems like it will be fun: building GDB on
> MacOSX with Python support enabled...

Keep in mind that on newer OS X releases, Apple no longer ships gcc or gdb.  The standard compiler is clang and it has its own debuggers.

>> For the python.org OS X binary installers, we go to a fair amount of
>> trouble to build Pythons that will work across a range of OS X
>> releases.  You might want to consider using one of them as a base.
>> It's usually a lot less work than trying to make it work yourself.
> 
> Hm, that's an idea.  I don't HAVE to build Python myself, actually, I
> just need (a) it to be relocatable, and (b) to add these extra modules
> to it so I can use it across systems without installing them
> individually by hand.


I'm not sure I know what you mean by "relocatable".  Like the third-party package manager solutions, the python.org installers place the Pythons they install in fixed locations (under /Library/Frameworks with links from /usr/local/bin) but one that does not conflict with any system files, including the Apple-supplied system Pythons.

I don't know exactly what you are trying to do but a couple of suggestions.  A number of widely-used Python projects, like Numpy, provide their own OS X installer package (.mpkg or .pkg) that installs into the site-packages directory of the python.org Python 2.7.  That's the sort of traditional way of doing it.

A more recent development is deploying Python packages (not Python itself) using the "wheel" format.  (http://wheel.readthedocs.org/en/latest/).  The wheel is a cross-platform format and replaces earlier binary distribution formats like eggs.  It is supported by recent versions of pip (https://pypi.python.org/pypi/pip), which itself replaces earlier installer commands like easy-install or using "python setup.py install".  Using pip and wheels have been endorsed by the Python core development team as the new one obvious way for distributions (PEP 453).  pip will be automatically included with Python 3.4 but it is very easy to install and use with current Pythons (3.3.x and 2.7.x).  You should be able to use pip and wheels, possibly platform-specific wheels, for deploying to all of your platforms: Linux, OS X, and Windows.  For OS X and Windows, the deployment would be very similar: install Python from a python.org installer, install pip (and its current setuptools dependency), install additional wheels.  On the various Linux systems, some of those steps could be replaced by using the system-supplied packages.  Also, if it is not desired or possible to install everything into a shared location on the deployed systems, e.g. there are multiple users or the user does not have admin privileges, you could either use the pip --user option or use virtual environments (virtualenv) in which to install wheels.

That may not help in your situation but those are the paths of least resistance these days.

--
  Ned Deily
  nad at acm.org -- []