tarfile's tar.extractfile() file-like object incompatible with pickle.load()?

[Tom B.]

"Matt Doucleff" <mattduke at ugcs.caltech.edu> wrote in message
news:3e6d6c21.0408271000.6007a361 at posting.google.com...
> "Tom B." <sbabbitt at commspeed.net> wrote in message
news:<1093572564.733113 at news.commspeed.net>...
> > "Matt Doucleff" <mattduke at ugcs.caltech.edu> wrote in message
> > news:3e6d6c21.0408261043.264c3a2d at posting.google.com...
> > > Hi everyone!  I must be doing something wrong here :)  I have a
> > > tarball that contains a single file whose contents are a pickled
> > > object.  I would like to unpickle the object directly from the tarball
> > > using the file-like object provided by extractfile().  Attempts to do
> > > this result in EOFError.  However if I first extract to a temporary
> > > file, then unpickle from there, it works.  The below code reproduces
> > > the problem (on my machine at least).  I'm running Python 2.3.4,
> > > manually installed on Debian Woody (original python removed).  Thanks!
> > >
> > > This sample code creates (and then removes) files in the tmp directory
> > > and in the current working directory.
> > >
> > > # demonstrates extractfile/unpickle failure (bug?)
> > >
> > > # pickle a dict to a temp file
> > > # create tar file, add temp file to it, close tar file
> > > # open tar file for reading
> > > # obtain file-like object for pickled file using extractfile()
> > > # attempt to unpickle dict from file-like object
> > > # fails with EOFError exception
> > >
> > > import tarfile
> > > import pickle
> > > import tempfile
> > > import os
> > >
> > > if __name__ == '__main__':
> > >     try:
> > >         hashtopickle = { 'a' : 1, 'b' : 2 }
> > >
> > >         # pickle to temp file
> > >         (fd, tmpfilename) = tempfile.mkstemp()
> > >         tmpfile = os.fdopen(fd, 'w')
> > >         pickle.dump(hashtopickle, tmpfile)
> > >         tmpfile.close()
> > >
> > >         # create tar; add temp file
> > >         tar = tarfile.open('tarpickle.tar', 'w')
> > >         tar.add(tmpfilename, 'pickledhash')
> > >         tar.close()
> > >
> > >         # remove temp file
> > >         os.remove(tmpfilename)
> > >
> > >         # open tarfile for reading, get filelike
> > >         tar = tarfile.open('tarpickle.tar', 'r')
>
> > >         filelike = tar.extractfile('pickledhash')
> > >
> > >         # fails
> > >         hashcopy = pickle.load(filelike)
> > >
> > >     finally:
> > >         # cleanup
> > >         os.remove('tarpickle.tar')
> >
> > It occurs to me that you need to do,
> >
> >  hashcopy = pickle.loads(filelike)
> >
> > if filelike is a string.
> >
> > Tom
> > P.S. have a look at pickle.dumps()
>
> The tarfile.extractfile() method does not read the contents
> of the encapsulated file into a string, but constructs a new
> object that implements file operations (it is like a file)
> and is intended to be used as if you had simply opened the
> tar-encapsulated file directly.
>
>     Matt

How about,

filelike = tar.extractfile('pickledhash')
filetext = filelike.read()
hashcopy = pickle.load(StringIO.StringIO(filetext ))

Tom