tarfile's tar.extractfile() file-like object incompatible with pickle.load()?
Tom B.
sbabbitt at commspeed.net
Fri Aug 27 16:19:23 EDT 2004
"Matt Doucleff" <mattduke at ugcs.caltech.edu> wrote in message
news:3e6d6c21.0408271000.6007a361 at posting.google.com...
> "Tom B." <sbabbitt at commspeed.net> wrote in message
news:<1093572564.733113 at news.commspeed.net>...
> > "Matt Doucleff" <mattduke at ugcs.caltech.edu> wrote in message
> > news:3e6d6c21.0408261043.264c3a2d at posting.google.com...
> > > Hi everyone! I must be doing something wrong here :) I have a
> > > tarball that contains a single file whose contents are a pickled
> > > object. I would like to unpickle the object directly from the tarball
> > > using the file-like object provided by extractfile(). Attempts to do
> > > this result in EOFError. However if I first extract to a temporary
> > > file, then unpickle from there, it works. The below code reproduces
> > > the problem (on my machine at least). I'm running Python 2.3.4,
> > > manually installed on Debian Woody (original python removed). Thanks!
> > >
> > > This sample code creates (and then removes) files in the tmp directory
> > > and in the current working directory.
> > >
> > > # demonstrates extractfile/unpickle failure (bug?)
> > >
> > > # pickle a dict to a temp file
> > > # create tar file, add temp file to it, close tar file
> > > # open tar file for reading
> > > # obtain file-like object for pickled file using extractfile()
> > > # attempt to unpickle dict from file-like object
> > > # fails with EOFError exception
> > >
> > > import tarfile
> > > import pickle
> > > import tempfile
> > > import os
> > >
> > > if __name__ == '__main__':
> > > try:
> > > hashtopickle = { 'a' : 1, 'b' : 2 }
> > >
> > > # pickle to temp file
> > > (fd, tmpfilename) = tempfile.mkstemp()
> > > tmpfile = os.fdopen(fd, 'w')
> > > pickle.dump(hashtopickle, tmpfile)
> > > tmpfile.close()
> > >
> > > # create tar; add temp file
> > > tar = tarfile.open('tarpickle.tar', 'w')
> > > tar.add(tmpfilename, 'pickledhash')
> > > tar.close()
> > >
> > > # remove temp file
> > > os.remove(tmpfilename)
> > >
> > > # open tarfile for reading, get filelike
> > > tar = tarfile.open('tarpickle.tar', 'r')
>
> > > filelike = tar.extractfile('pickledhash')
> > >
> > > # fails
> > > hashcopy = pickle.load(filelike)
> > >
> > > finally:
> > > # cleanup
> > > os.remove('tarpickle.tar')
> >
> > It occurs to me that you need to do,
> >
> > hashcopy = pickle.loads(filelike)
> >
> > if filelike is a string.
> >
> > Tom
> > P.S. have a look at pickle.dumps()
>
> The tarfile.extractfile() method does not read the contents
> of the encapsulated file into a string, but constructs a new
> object that implements file operations (it is like a file)
> and is intended to be used as if you had simply opened the
> tar-encapsulated file directly.
>
> Matt
How about,
filelike = tar.extractfile('pickledhash')
filetext = filelike.read()
hashcopy = pickle.load(StringIO.StringIO(filetext ))
Tom
More information about the Python-list
mailing list