Does hashlib support a file mode?

[Anssi Saari]

Phlip <phlip2005 at gmail.com> writes:

> If the file were huge, the file.read() would allocate a big string and
> thrash memory. (Yes, in 2011 that's still a problem, because these
> files could be movies and whatnot.)

I did a crc32 calculator like that and actually ran into some kind of
string length limit with large files. So I switched to 4k blocks and
the speed is about the same as a C implementation in the program
cksfv. Well, of course crc32 is usually done with a table lookup, so
it's always fast.

I just picked 4k, since it's the page size in x86 systems and also a
common block size for file systems. Seems to be big enough.
io.DEFAULT_BUFFER_SIZE is 8k here. I suppose using that would be the
proper way.