[Numpy-discussion] checksum on numpy float array

[josef.pktd at gmail.com]

On Thu, Dec 4, 2008 at 6:17 PM, Brennan Williams
<brennan.williams at visualreservoir.com> wrote:
> My app reads in one or more float arrays from a binary file.
>
> Sometimes due to network timeouts etc the array is not read correctly.
>
> What would be the best way of checking the validity of the data?
>
> Would some sort of checksum approach be a good idea?
> Would that work with an array of floating point values?
> Or are checksums more for int,byte,string type data?
>

If you want to verify the file itself, then python provides several
more or less secure checksums, my experience was that zlib.crc32 was
pretty fast on moderate file sizes. crc32 is common inside archive
files and for binary newsgroups. If you have large files transported
over the network, e.g. GB size, I would work with par2 repair files,
which verifies and repairs at the same time.

Josef