pickle.load() extremely slow performance
John Machin
sjmachin at lexicon.net
Fri Mar 20 18:40:25 EDT 2009
On Mar 21, 9:25 am, Jim Garrison <j... at acm.org> wrote:
> I'm converting a Perl system to Python, and have run into a severe
> performance problem with pickle.
>
> One facet of the system involves scanning and loading into memory a
> couple of parallel directory trees containing OTO 10^4 files. The
> trees don't change during development/testing and the scan takes 30-40
> seconds, so to save time I cache the loaded tree structure to disk, in
> Perl with module Storable, and in Python with pickle.
>
> In Perl, the save operation produces a file of about 3MB, and both
> save and restore take a second or two. In Python, pickle.dump()
> produces a similar-size file but takes 20 seconds, and pickle.load()
> takes 45 seconds, which is actually LONGER than the time required to
> scan the directory trees.
>
> Is there anything I can do to speed up pickle.load() to get
> performance comparable to Perl's Storable?
Have you read this:
http://www.python.org/doc/2.6/library/pickle.html
?
Have you considered using cPickle instead of pickle?
Have you considered using *ickle.dump(..., protocol=-1) ?
More information about the Python-list
mailing list