Can't load .pyd files from embedded Python interp

[Edward K. Ream]

Hi,

I'm having difficulty importing modules defined in .pyd files from an
embedded Python interpreter.  The problem seems to be that two sets of
variables are being used and no linking is happening when the DLL is
loaded.  Perhaps you DLL gurus will spot the problem immediatly. 
Details follow...

I've embedded the Python interpreter in a wxWindows test app using the
Borland compiler. The interpreter was embedded in a straightforward way,
by creating python15.lib and parserLib.lib from the sources, then
linking these libs with no special options into my app.

I've also used the python15.lib and parser.lib files from the Windows
distribution, with approximately the same results.  Because of problems
with FILE * params in the C API, it seemed best to build the .lib files
using the Borland compiler.

The app can execute code such as the following from the
wxWindows OnInit code:

    Py_Initialize();
    PyRun_SimpleString("for i in sys.path: print i");

Py_Initialize() exercises lots of code, including all the code I
modified for the Borland compiler, so most everything is working.

However, I am not able to import files defined in any .pyd file, say
like this:

    Py_Initialize();
    PyRun_SimpleString("import parser") ;

where parser is contained in parser.pyd.  No matter what .pyd file is
imported, the code fails by calling Py_FatalError inside the DLL's
initialization routine, parser.initparser in this example.  What happens
is this:

1. The call to PyRun_SimpleString("import parser") eventually results in
a call to _PyImport_LoadDynamicModule which loads the DLL called
parser.pyd by calling:

    hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH);

The call succeeds.

2.  _PyImport_LoadDynamicModule then calls:

    p = GetProcAddress(hDLL, funcname);

where funcname is "initparser" in this example.  This call also
succeeds, so p points to the init function for parser module.

3.  _PyImport_LoadDynamicModule then calls:

        (*p)();

This statement calls the init function for the imported .pyd file.

4.  The init function appears to be entered correctly and very soon the
init function calls Py_FatalError.  The reason is clear:  the init code
is using a separate copy of the variable PyThreadState_Current and that
variable is NULL because it has never been initialized.

The Borland debugger shows this variable as
[python15.PyThreadState_Current] rather than [interp_head+4] for the
same variable in the embedded interpreter.

So what is happening seems clear: there are two separate variables
called PyThreadState_Current, and the call to Py_Initialize() in the
embedded interpreter initializes only the interpreter's copy, not the
DLL's copy.

In my mind there are several possible ways to make the two copies of
PyThreadState_Current refer to the same memory.

1.  [Unlikely]  With some special initialization code before calling
(*p)().  This is unlikely for several reasons, of which one is there is
no indication anywhere that importing .pyd files should require special
code.  And changing code in _PyImport_LoadDynamicModule seems out of the
question.

2.  [Unlikely] Perhaps the call (*p)() is defective.  Borland C has
several directives for declaring function pointers.  The present Borland
code declares p with:

    typedef __cdecl FARPROC dl_funcptr;
    dl_funcptr p;

However, the difference between __cdecl, __pascal and __stdcall should
make a diffence only if p has arguments.  And the init routine seems to
be called properly.

3.  [More likely?]  We should provide information to LoadLibraryEx so
that LoadLibraryEx can fixup references within the DLL to point at the
corresponding locations in the embedded interpreter.  I've looked in
vain for information about how to do this.  Is this a matter of using
special link options, or of providing tables to LoadLibraryEx?

4.  [Less likely?] The Windows 32-bit API docs mention the DllEntryPoint
mechanism.  I don't see anything in the _PyImport_LoadDynamicModule to
indicate that this mechanism is used.  Instead, .pyd files seem to be
using their own kind of init function.

After all this background, can anyone tell me how to load .pyd files
from an embedded Python interpreter?  Also, can anyone suggest a URL
that really explains Windows DLL files?  The entries at
http://msdn.microsoft.com/library
seem less than completely satisfactory.

Edward
--------------------------------------------------------------------
Edward K. Ream   email:  edream at tds.net
Leo: Literate Editor with Outlines
Leo: http://personalpages.tds.net/~edream/front.html
--------------------------------------------------------------------