[CentralOH] Incrementally Downloading ISH Data

[jep200404 at columbus.rr.com]

Hi Eric, 

Rehashing what we discussed, plus one new thought.

FTP probably can not send a file beginning from an arbitrary point.
wget can.

Recent versions of gzip are deterministic, 
so if only the tail of an uncompressed file changes, 
then only the tail of a compressed file changes.

However, if there is a change other than at the end, 
whole compressed file changes. This happens, albeit rarely. 
Would need to reload a bit more than just the tail end 
to detect such rare changes.

I think rsync tends to resend whole file if any part of 
it changes. I'm not sure about rsync. 

Bittorrent protocol can pick up where it left off, 
and could retransmit only the 64 KByte chunks that 
have changed. So everyday we could easliy download 
just the changes. That would require a change on NWS server. 
The hard part would probably be getting them to change their 
server. Sales pitch could be that it would reduce the load 
on their servers, both my allowing incremental downloads, 
and by other peers sharing the load. We would need to test 
the ability for bittorrent to only download changes before 
pitching it.

Jim