PyImport_Import() fails - what am I doing wrong here?

[Bob Hepple]

On Tue, 25 Feb 2003 06:59:53 GMT
Jim <jbublitzNO at SPAMnwinternet.com> wrote:
> 
> I'm not real expert on this, but since nobody else has
> answered, I'll give it a try. It sounds like the problem
> might be with the way you're linking the embedded Python
> interpreter - it needs to link with -Wl,E or if using dlopen
> it needs to be loaded with RTLD_GLOBAL set. There's a way
> to do the equivalent with libtool, but I'm not sure what
> that is. I do it in KDE, which has a 'load library global'
> method built-in. This is for linking *libpython* to your
> app, not for linking libfoobar.
> 
> The symptoms would be that you could import sys or os with
> the embedded interpreter because they're built-in, but not
> something like the Python 'time' module or your own module.
> 
> 

Thanks for taking a stab at this - it really is not easy.

I have tried the -Wl,-E option but it doesn't fix the problem. On reading
the doco for ld(1), that option makes objects in the main program available
to be called by shared objects. This is the opposite of what I want. I
want the embedded python interpreter (libpython1.5.a - a static object) to
pull in and link a shared library. In turn, that shared library needs to
link in an additional shared library but definitely not anything from the
main program.

Perhaps I could assay a little more explanation ...

What I need to do is to create a test harness using some existing
software. I anonymised it to `foobar' to avoid introducing distracting
details in something already hard to explain - in fact it's a pkcs#11
implementation, sometimes called cryptoki.

Our test package is in C. It exercises every one of 83 API calls and is a
mature piece of software. I want to create a python binding for all these
calls and use the existing testing C program to exercise it. I therefore
need a c->python layer as well as the python->c layer. Here is the structure

The c program:

test.c calls foobar()
/usr/lib/libfoobar.so: - dynamically linked

cc -o test test.c -lfoobar

Too easy!

Now the python->c way of doing it:

python test.py - calls foobar() extension:
/usr/lib/python1.5/site-packages/foobar.so - dynamically linked (probably by dlopen in python)

Also, too easy!


Now for the c->python->c version of the test:

./test.c - calls foobar() statically linked from:
./foobar.c - a shim which calls the python interpreter:
/usr/lib/python1.5/config/libpython1.5.a - the python interpreter, calls:
/usr/lib/python1.5/site-packages/foobar.so - the python extension, calls foobar() from:
/usr/lib/libfoobar.so - dynamically linked (probably by dlopen in python)

test.c calls foobar() as before - in fact, it's the same test.o. foobar()
is resolved at _link_ time (a static link) to the shim routine
python_foobar.c:foobar(). The program is also linked against the static
library /usr/lib/python1.5/config/libpython1.5.a. As such, it has no
undefined externals that need resolving at runtime (apart from libc et
al).

At runtime, this shim makes a call to the python interpreter which then
presumably uses dlopen() to make a call to the python extension foobar()
which lives in /usr/lib/python1.5/site-packages/foobar.so. This is the
step that fails - for some reason Py_Import_Import() fails silently.

Finally, the python interpreter needs to resolve against the dynamic
library /usr/lib/libfoobar.so to make the final call to the real foobar()

cc -o test test.c python_foobar.c -lfoobar -L/usr/lib/python1.5/config -lpython1.5

Is there, for instance, some way to make Py_Import_Import() fail a bit
more verbosely? Some environment parameters?

Maybe I need to recompile the libpython.a with debug option on to see what
is happening.



-- 
Bob Hepple, Research & Development Group
Eracom Technologies Australia Pty. Ltd.
28 Greg Chappell Drive, Burleigh Heads, Qld. 4220, Australia
Tel.: +61 7 5593 4911               Fax.: +61 7 5593 4388
mailto:bob.hepple at eracom.com.au http://www.eracom-tech.com