cPickle.load vs. file.read+cPickle.loads on large binary files
Nagy László Zsolt
gandalf at shopzeus.com
Wed Nov 18 04:00:21 EST 2015
> Perhaps there is a size threshold? You could experiment with different block
> sizes in the following f.read() replacement:
>
> def read_chunked(f, size=2**20):
> read = functools.partial(f.read, size)
> return "".join(iter(read, ""))
>
Under win32 platform, my experience is that the fastest way to read
binary file from disk is the mmap module. You should try that too.
More information about the Python-list
mailing list