PyImport_Import() fails - what am I doing wrong here?

[Bob Hepple]

On Tue, 25 Feb 2003 10:16:20 +1000
Bob Hepple <bhepple at freeshell.org> wrote:

> I have a Python extension module (foobar) written in C. I want to use
> it as a python extension both in python scripts and in an embedded C
> app.
> 

Well I fixed this in an all-nighter so having posed the question I'll
answer it myself as it might save someone else the agony.

The problem is caused by the fact that the python interpreter is only
available as a static library - this gives problems in resolving symbols
in a shared library extension when the -Wl,-E option is not possible eg.
when the shared library extension has some symbols in common with the
main.

The fix was to link the shared library extension with -lpython when the
shared library is built - just add this to your setup.py when building the
extension shared lib:

	library_dirs = ["/your/fav/lib", "/usr/lib/python1.5/config"],
	libraries = ["foobar", "python1.5"])])

I'm grateful for Jim Bublitz for pointing out the fact that python needs
-Wl,-E. Incidently, the undocumented ld(1) option -rdynamic appears to do
the same thing and that's what setup.py suggested as the right option to
use. Anyway, it fails in the same way for me.

I _think_ that if python were available as a shared library then the
runtime linker would have succeeded. Certainly, when these shenanigans are
played using java and jvm, there is no muss or fuss in linking - a simple
straight link with the shared libraries finds everything as expected and
also shields the main program symbols from the shared library. I believe
this is because the jvm is a shared object and I consider this lack a
defect in python. If anyone is voting for a shared library version of
python then I'll add my vote.

The lack of a shared python library is no different in 2.2 - I've yet to
investigate 2.3 or the history in this group of why there is no such
shared library  - but if anyone wants to comment please jump right in!

The other nuisance factor was the critical error condition not propagating
properly in the python interpreter around the dlopen() call. I have only
looked at 1.5 so this may have been fixed by now (probably not in 2.2 as
the same symptom is present) but in Python/importdl.c:439 we have:

#ifdef RTLD_NOW
		/* RTLD_NOW: resolve externals now
		   (i.e. core dump now if some are missing) */
		void *handle = dlopen(pathname, RTLD_NOW);
#else
		void *handle;
		if (Py_VerboseFlag)
			printf("dlopen(\"%s\", %d);\n", pathname,
			       RTLD_LAZY);
		handle = dlopen(pathname, RTLD_LAZY);
#endif /* RTLD_NOW */
		if (handle == NULL) {
			PyErr_SetString(PyExc_ImportError, dlerror());
			return NULL;
		}

On linux, RTLD_NOW is defined but it does not, in fact, drop a core if
externals are not resolved, it returns handle = 0. For some reason which I
could not determine, the dlerror() message was never printed - but if I
put a puts(dlerror()) at that point I could see the linker error message -
it was failing to find various Py_ entrypoints for the shared lib.

I only found the error and the message my re-building libpython with -g
and a gdb trace through it which is a rather extreme way to have to see
what's going on!

Can anyone sus why the error failed to propagate?


Cheers


Bob


-- 
Bob Hepple
mailto:bhepple at freeshell.org http://bhepple@freeshell.org