Hi reliability files, writing,reading,maintaining

[Martin P. Hellwig]

John Pote wrote:
<cut>
> So my request:
> 1. Are there any python modules 'out there' that might help in securely
> writing such files.
> 2. Can anyone suggest a book or two on this kind of file management. (These
> kind of problems must have been solved in the financial world many times).
> 
<cut>
I can't answer your specific questions but I got the feeling that you're 
  barking at the wrong tree ;-)

You don't want to solve this in you application, file management is what 
the OS and hardware is about. "Military" grade solutions are often 
(depending on their criticalness) double or triple hot spares in the 
same room which can be seen as a "unit" and these units are duplicated 
on remote locations (at least 25km of each other) syncing their data 
with standard tools like rsync.

If you don't have military budget, I would suggest to do it a little 
less expensive, like having a couple of Solaris machines (three or four 
will do it) in the same room, using a part of their diskspace for ZFS.

Then let your application write your data to that ZFS partition and if 
you are particular paranoid you can build in a checksum that can be 
calculated by other machines without the need for the original received 
data (ZFS has a built-in mechanism for that so you might just want to 
call that).

There is nothing wrong for assuming a certain level of hardware, well at 
least not if its very clearly communicated to all parties.
ZFS is open source (by SUN) but currently only implemented in Solaris, 
which you also can (commercially) use without charge.

Now you only got to figure out how to implement a heartbeat mechanism 
between you fail-over applications :-)

-- 
mph