pickle.load() extremely slow performance

[John Machin]

On Mar 21, 9:25 am, Jim Garrison <j... at acm.org> wrote:
> I'm converting a Perl system to Python, and have run into a severe
> performance problem with pickle.
>
> One facet of the system involves scanning and loading into memory a
> couple of parallel directory trees containing OTO 10^4 files.  The
> trees don't change during development/testing and the scan takes 30-40
> seconds, so to save time I cache the loaded tree structure to disk, in
> Perl with module Storable, and in Python with pickle.
>
> In Perl, the save operation produces a file of about 3MB, and both
> save and restore take a second or two.  In Python, pickle.dump()
> produces a similar-size file but takes 20 seconds, and pickle.load()
> takes 45 seconds, which is actually LONGER than the time required to
> scan the directory trees.
>
> Is there anything I can do to speed up pickle.load() to get
> performance comparable to Perl's Storable?