PyImport_Import() fails - what am I doing wrong here?

[Jim]

Bob Hepple wrote:
> Apologies for re-posting this but I must have obfuscated it badly the
> first time. Please allow me to re-state the problem - this is driving
> me nuts.

> I have a Python extension module (foobar) written in C. I want to use
> it as a python extension both in python scripts and in an embedded C
> app.

> It runs fine when called from python:

> #!/usr/bin/python
> import foobar
> foobar.init()

> All seems peachy. 

> However when I try to call it from an embedded python interpreter in a C
> program, the PyImport_Import() method fails:

> #include <Python.h>
> ...
>     Py_Initialize();
>     pName = PyString_FromString("foobar");
> 	if (!pName) {
> 		retval = CKR_GENERAL_ERROR;
> 		goto exit_point;
> 	}
> 
>     pModule = PyImport_Import(pName);
> 
> ... returns NULL
> 
> I tried PyImport_ImportModule("foobar") too with the same result.

> The initfoobar() method in the extension module never gets called (I
> put a printf() call there to check). There are no error messages from
> ld or python.

> If I run 'strace' on the embedded version, I can see something
> (presumably dlopen()) opening the extension module and (presumably a
> linker) opening the shared library: ...

> open("/usr/lib/python1.5/site-packages/foobar.so", O_RDONLY) = 3
> fstat64(3, {st_mode=S_IFREG|0775, st_size=111607, ...}) = 0
> open("/usr/lib/python1.5/site-packages/foobar.so", O_RDONLY) = 4
> read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@\23\0\000"...,
> 1024) = 1024
> ...
> close(4)                                = 0
> open("/usr/local/lib/libfoobar.so", O_RDONLY) = 4
> read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360d\1"...,
> 1024) = 1024
> fstat64(4, {st_mode=S_IFREG|0775, st_size=2204306, ...}) = 0
> 
> Oddly, the foobar extension package is opened twice - presumably the
> first time it is loading the modules and the second time it is the
> linker trying to resolve externals. These are actually resolved later
> when /usr/local/lib/libfoobar.so is opened.

> The module foobar lives in /usr/lib/python1.5/site-packages/foobar.so
> and is built by setup.py in the usual way from foobar.c and a -l
> linker reference to libfoobar.so which itself lives in /usr/local/lib:

> running install
> running build
> running build_ext
> building 'foobar' extension
> gcc -g -O2 -fPIC -I/usr/local/include -I/usr/include/python1.5 -c foobar.c -o build/temp.linux-i686-1.5/foobar.o
> gcc -shared build/temp.linux-i686-1.5/foobar.o -L/usr/local/lib -lfoobar -o build/lib.linux-i686-1.5/foobar.so
> running install_lib
> copying build/lib.linux-i686-1.5/foobar.so ->
> /usr/lib/python1.5/site-packages

> Puzzling, to say the least? Any ideas out there?

> Background - Linux RedHat 7.3 / python 1.5 (2.2 makes no difference)

I'm not real expert on this, but since nobody else has
answered, I'll give it a try. It sounds like the problem
might be with the way you're linking the embedded Python
interpreter - it needs to link with -Wl,E or if using dlopen
it needs to be loaded with RTLD_GLOBAL set. There's a way
to do the equivalent with libtool, but I'm not sure what
that is. I do it in KDE, which has a 'load library global'
method built-in. This is for linking *libpython* to your
app, not for linking libfoobar.

The symptoms would be that you could import sys or os with
the embedded interpreter because they're built-in, but not
something like the Python 'time' module or your own module.


Jim