app runs fine with interpreter, but not under py2exe

[Doug Morse]

Hi Harald,

Bond here, James Bond.  I accepted your mission. :)

Unfortunately, the mission failed.  Creating a "testapp.py" as you described
(i.e., with the suggested import statements) runs just fine with the
interpreter.  again, however, py2exe does the same thing as the original
problem -- that is, copies numpy.core's versions of multiarray.pyd and
umath.pyd into dist/, creates what look like stub multiarray.pyc and umath.pyc
files in library.zip/ and library.zip/numpy/core/.  when this py2exe version
is run, it throws the same (original) TypeError: data type not understood
error.

Also, with either my original program or testapp.py, I cannot seem to modify
library.zip in such a way as to get the program to run.  Basically, I unzipped
library.zip, made the same modifications to it's contents as I did when using
the --skip-archive option (i.e., remove the stub files and put the right .pyd
files in the right locations) and then recreated the library.zip file.
Afterwards, running either origapp.exe or testapp.exe, the program bombs out
with ImportError: no module named multiarray.

In addition, I tried using py2exe to make a simple "Hello, world" program.  It
runs just fine.  Then, I "zip -m library.zip *.pyd" in the dist/ directory to
move the .pyd files into library.zip.  Then, running hello.exe again works
just fine.  However, if I add "import bz2; print bz2.__author__" after "print
'Hello, world!'" (the only line in hello.py), hello.exe bombs out with
"ImportError: DLL load failed: The specified module could not be found" when I
run it after having moved *.pyd into library.zip/. (bz2 was one of only two
.pyd files that py2exe put into dist/.)

So, I can seem to figure out how to get .pyd files into library.zip, even when
there are no name conflicts.  What am I doing wrong?

Finally, regarding the stub files, if I use vi to edit the py2exe-generated,
536-byte bz2.pyc file (there is no such file in my python install directory
hierarchy), I see the word "import" very close to the word "bz2.pyd".  This
suggests to me that py2exe might indeed be creating simple import statements
in a short .py file for each .pyd file and then compiling it into a .pyc stub
file -- or at least doing something similar.  Of course I can't be sure of
this, but it seems likely to me.

Thanks again to you and everyone.  I'll definitely visit the py2exe wiki and
see what I can come up with (and or course will report back with success /
failures).  If you or anyone has any further thoughts based on this post, I'm
all ears and would be most grateful.

Regards,
Doug


On Sun, 16 Mar 2008 00:33:31 -0700 (PDT), GHUM <haraldarminmassa at gmail.com>
wrote:
>  Doug,
> 
> > as I quickly noticed that "library.zip" does NOT contain ANY .pyd files.
> > I'm guessing that they can't be in library.zip for a reason (i.e., they are
> > DLL files, essentially, and thus must be readily available to be loaded
> > into memory).
> 
>  .dll and .pyd files CAN be within library.zip and even within the
>  single-file-exe.
>
>  There is some Hellermagic within _memimporter.pyd that loads the .dll
>  and .pyd from a zipfile. When you scan the build_exe.py, you will
>  explicitely find:
> 
>          print "*** finding dlls needed ***"
>          dlls = self.find_dlls(extensions)
>          self.plat_finalize(mf.modules, py_files, extensions, dlls)
>          dlls = [item for item in dlls
>                  if os.path.basename(item).lower() not in
>  self.dll_excludes]
>          # should we filter self.other_depends in the same way?
> 
> > Is there a work-around for this, then?
>      -rwxr-xr-x 1 morse None  26624 Nov 28  2006 dist/multiarray.pyd
>      -rwxr-xr-x 1 morse None 348160 Nov  8 16:16 dist/numpy/core/
>  multiarray.pyd
>      -rwxr-xr-x 1 morse None 192512 Nov  8 16:16 dist/numpy/core/
>  umath.pyd
>      -rwxr-xr-x 1 morse None  54272 Nov 28  2006 dist/umath.pyd
> 
>  let's try: your mission is, should you chose to accept it, to make a
>  py2exed application to load two different multiarray.pyd, who are on
>  the visible level only different by their place in the file system.
> 
>  So, please make a minimal script:
> 
>  import multiarray
>  import numpy.core.multiarray as ncmultiarray
> 
>  and, using Peters test
> >>> from multiarray import zeros
>  assert zeros((1,), "1") == array([0], '1')
> 
>  and check, if those multiarrays are the correct ones. If yes, you can
>  try to build upon this by making all imports explicit as described
>  above. (Yes, I know, that's not necessary in plain python ... but it
>  saved me in some weird "import from .zip" and "import from database"
>  situations.)
> 
> > (a) tell py2exe how to *correctly* handle multiple multiarray.pyd and
> > umath.pyd files
> 
>  somewhere deep inside I have a BAAAD feeling about to files having the
>  same name but very very different content - but that's not your call
>  in this situation.
>  Are you able to build that libraries yourself? Then you could move
>  down and build the multiarray.pyd as ncmultiarray.pyd and adjust all
>  imports accordingly. (lots of work)
> 
> > manual create the "stub"-like .pyc files that py2exe creates to point to
> > these alternate .pyd files and then place these stubs in
> > library.zip/numpy/core?  
> 
>  As much as I know there is only one central "loader stub", which
>  ensures that .dlls get imported from memory, see above. py2exe does
>  not create something like p2elma.py containing "import multiarray.pyd"
> 
>  best luck, and please take your time to scan the py2exe.org wiki,
>  there are many similiar challanges with various packages, maybe you
>  get another workaround-idea (and, if we find one, PLEASE put it down
>  there for others to learn)
> 
>  best wishes,
> 
>  Harald