[python-win32] mxDateTime problem with embedded Python

[Mark Hammond]

On 25/08/2012 11:28 AM, Will Sadkin wrote:
> On 24/08/2012 4:00 AM, Will Sadkin wrote:
>>> Hi all,
>>>
>>> I have a bit of a stumper.  We have a C++ program in which we embed a
>
>>> release python interpreter, so that both our C++ and python programs
>>> can share a single codebase, written in Python.  When building the
>>> program for debug, we always #undef _DEBUG while including Python.h,
>>> and always therefore link with python27.lib, and embed the *release*
> interpreter.
>
>> There are other symbols - it looks like you want to ensure DEBUG also
> isn't set and that NDEBUG is.
>
> I tried this, but it made no difference.
>
>>> (We did this so that we don't have to locate and/or build debug
>>> versions of the python interpreteter or extension dlls we may load
> (eg
>>> pyodbc, mx, etc.  At least that was the theory.))
>>>
>>> This embedded interpreter ends up importing a python package that
>>> tries to run the line:
>>>
>>>       from mx import DateTime
>>>
>>>  From within the release C++ executable, this works just fine.  But
>>> with the debug executable, it generates the following exception on
> that line:
>>>
>>>       DLL load failed: A dynamic link library (DLL) initialization
>>> routine failed.
>>>
>>> I have a hypothesis as to why this is happening, bug am not sure how
>>> to confirm it, nor what to do about it.
>>> I suspect that the python27.dll is a statically linked dll,
>>
>> I'm not sure what you mean here - if you are talking about the linkage
> of the
>> CRT, then it is not static - Python uses the DLL version of the CRT.
> You should be able
>> to confirm this with depends.exe, and also by looking at the "output"
> tab of MSVC -
>> it should report the CRT being loaded as python27.dll loads.
>
> Ok, here is what the output window of the attached debugger says as the
> pyd file
> loads (I'm running a 32bit version of python27 on my 64bit laptop, which
> is why the
> paths all have SysWOW64 and x86 in them, but normally this is running on
> a 32bit machine.):
>
> 'BuildServ.exe': Loaded 'C:\Program Files
> (x86)\Python27\Lib\site-packages\mx\DateTime\mxDateTime\mxDateTime.pyd',
> Binary was not built with debug information.
> 'BuildServ.exe': Loaded 'C:\Windows\SysWOW64\msvcr90.dll', Symbols
> loaded.
> 'BuildServ.exe': Unloaded 'C:\Program Files
> (x86)\Python27\Lib\site-packages\mx\DateTime\mxDateTime\mxDateTime.pyd'
> 'BuildServ.exe': Unloaded 'C:\Windows\SysWOW64\msvcr90.dll'
>
> And yet the pyd import generates the DLL load failure above.
>
> [...]
>> That shouldn't be a problem - if all Python components are linked
> against the
>> same CRT, life should be OK even if the embedding component uses a
> different CRT.
>> Unless, of course, that embedding component attempts to directly call
> Python...
>
> I'm pretty sure I'm not, which leaves me completely mystified as to why
> mxDateTime won't
> Initialize properly in my debug executable...
>
>> Also note that it might be useful to simply change your release build
> to not optimize
>> and to include debug symbols - that should let you step through the
> release version
>> and avoids mixing-and-matching CRT versions.
>
> I had tried this before with no success, but I took your advice, and
> finally figured out
> how to get symbols and source code access for my release executable, and
> then tracked down
> my other bug.  Your suggestion makes me wonder what the point of debug
> executables at all
> if one can do this, but assuming that there are somethings you can do
> with a debug exe that
> you can't with a release exe, I'd still like to know why the debug exe
> can't load mxDateTime!

I just realized that is going to be due to the joy of manifests.  The 
Python DLL goes to some effort so that .pyd modules don't need to 
declare in their manifest that they use the CRT - however, that only 
works when the pyd uses the same CRT as Python itself.  Python is almost 
certainly using the debug CRT (but as your copy of the output from the 
debugger didn't show the Python DLL loading, I can't be 100% sure of 
that), hence mxDateTime attempting to use that CRT without having it 
declared in the manifest failed.

The manifest story sucks - fortunately Python 3.3 and later will not 
suffer that problem as VS2010 dropped the requirement for the CRT to be 
an "assembly" and hence requiring an entry in the manifest.

Mark