py2exe + svn - the final drama

[David Bolen]

Timothy Smith <timothy at open-networks.net> writes:

> Timothy Smith wrote:
(...)
> >zipimport.ZipImportError: bad local file header in Z:\temp\library.zip
> >
> > not that once i have finished client.update(''), it has successfully
> > updated the zipfile, i open a dialoge box saying "click ok and
> > restart program" AFTER i click on the above error pops up and my app
> > shuts down as intended.
> >
> >ideas?
> >
> ok i have done some digging and i cound this
> 
> /* Check to make sure the local file header is correct */
> 	fseek(fp, file_offset, 0);
> 	l = PyMarshal_ReadLongFromFile(fp);
> 	if (l != 0x04034B50) {
> 		/* Bad: Local File Header */
> 		PyErr_Format(ZipImportError,
> 			     "bad local file header in %s",
> 			     archive);
> 		fclose(fp);
> 
> 
> can anyone explain to me about zip file headers and why it would be
> different/incorrect and give me this error?

Are you perhaps trying to update the zip file in-place while it is still
being used by the application?  I'm not sure that's a safe operation.  A
quick peek at the same module where I think you found the above code shows
that when a zip importer instance is associated with a zip file, the
directory for that zip file is read in and cached.  So the importer is
holding onto offset information for each file based on the contents of the
zip directory at initialization time.

If you then change the file contents (such as updating it with svn), those
offsets will no longer be valid.  I then expect that during your process
exit, some bit of code is performing an extra import, which accesses the
wrong (based on the new file contents) portion of the zip file, and the
above safety check prevents it from loading an erroneous set of bytes
thinking its a valid module.

I expect you need to work on a mechanism to update the file
independently of the running copy, and then arrange to have it moved
into place for a subsequent execution.  Or find some way to have the
zip importer refresh its directory information or make a new importer
instance once the zip file is updated.

One (untested) thought ... before the update, make a copy of your
current library.zip as some other name, and adjust your sys.path to
reference that name (rather than the default pointer to the main
library.zip that py2exe initializes things with).  That should force
any future imports to access the old copy of the zip file and not the
one that svn will be updating.  Since you need to leave that zip file
copy in place through the exit (to satisfy any trailing imports),
arrange for your application to check for that copy on startup and
remove it if present.

Or, after looking through import.c handling for zip file imports,
there might be a simpler way.  ZIP imports are handled by a
zipimporter installed in sys.path_hooks, and once a specific path
element has a path hook instantiated for it (based on the sys.path
element name) it is cached in sys.path_hooks_cache.

So, simply clearing out the path_hooks_cache entry for your main
library.zip file should cause the next import attempt to re-create a
new zipimporter instance and thus re-open the file and re-load the
directory information.

I don't know if py2exe installs the library.zip into sys.path just as
"library.zip" or with some path information, but try checking out the
keys in sys.path_hooks_cache from your application when it is running.
You should find an entry (probably the only one unless you explicitly
augment sys.path yourself) for library.zip - clear out that key after
the update and see how it works.

Heck, since you're the efficiency hit is likely not an issue, just
flush all of sys.path_hooks_cache and don't even worry about the
actual key name for library.zip.  So a simple:
    sys.path_importer_cache.clear()
call after your update completes may do the trick.

-- David

PS: In the same way that updating the library.zip under the running
application is tricky, you might run into issues if you end up trying
to update one of the extension modules.  svn might not be able to
update it (depending on how it deals with "in use" files).