Hi reliability files, writing,reading,maintaining

[John Pote]

Thanks for all the replies,

> <cut from Cameron Laird>
> Also, the whole idea of basing reliability on HTTP uploads of 50 bytes
> at a time sounds to me a bit ... strained.  There *must* be simpler
> ways--and simpler goes a long way in the direction of trustworthy.
>

The motivation to look at http: is the widespread avaiability of internet 
connections and standard servers able to run CGI scripts. In particular the 
availability of low cost GPRS modems (solar panel and/or wind gen) is very 
attractive for remote locations where there is  no power, telephone line or 
cable company.

However, I take your point about HTTP uploads of 50 or so bytes is a little 
overhead heavy. Recently I've come to know of reasonably priced fully 
managed dedicated servers in secure buildings. So I am now thinking of a 
direct TCP/IP port connection. I know Python can do this as I found the 
appropriate standard modules and set up a simple proving system on my 
private network (one port listener, one sender - no problem!). Are there 
security issues in this approach?

I realise http: reliability is not a great as it might be but we have a 
reasonable amount of time for re-tries (an hour or two) if some packets get 
lost. My recent experience of telneting to our server in London (I'm in the 
midlands) is quite revealing. Round trip time is generally sub .5 sec. Makes 
me think all that waiting for web pages is the server rather than the 
internet itself.

Any thoughts on *simpler* ways to achieve my goal of remote data to web 
servers greatly appriciated.

It would still be useful to have anyones thoughts on securely writing the 
data to disk files. The off the shelf "military" approach suggested by 
Martin Hellwig is outside our budget.

Thanks again.

John

"Cameron Laird" <claird at lairds.us> wrote in message 
news:ktpsb3-avr.ln1 at lairds.us...
> In article <43eb4480$0$11064$e4fe514c at news.xs4all.nl>,
> Martin P. Hellwig <mhellwig at xs4all.nl> wrote:
>>John Pote wrote:
>><cut>
>>> So my request:
>>> 1. Are there any python modules 'out there' that might help in securely
>>> writing such files.
>>> 2. Can anyone suggest a book or two on this kind of file management. 
>>> (These
>>> kind of problems must have been solved in the financial world many 
>>> times).
>>>
>><cut>
>>I can't answer your specific questions but I got the feeling that you're
>>  barking at the wrong tree ;-)
>>
>>You don't want to solve this in you application, file management is what
>>the OS and hardware is about. "Military" grade solutions are often
>>(depending on their criticalness) double or triple hot spares in the
>>same room which can be seen as a "unit" and these units are duplicated
>>on remote locations (at least 25km of each other) syncing their data
>>with standard tools like rsync.
>>
>>If you don't have military budget, I would suggest to do it a little
>>less expensive, like having a couple of Solaris machines (three or four
>>will do it) in the same room, using a part of their diskspace for ZFS.
>>
>>Then let your application write your data to that ZFS partition and if
>>you are particular paranoid you can build in a checksum that can be
>>calculated by other machines without the need for the original received
>>data (ZFS has a built-in mechanism for that so you might just want to
>>call that).
>>
>>There is nothing wrong for assuming a certain level of hardware, well at
>>least not if its very clearly communicated to all parties.
>>ZFS is open source (by SUN) but currently only implemented in Solaris,
>>which you also can (commercially) use without charge.
>>
>>Now you only got to figure out how to implement a heartbeat mechanism
>>between you fail-over applications :-)
> .
> .
> .
> Also, the whole idea of basing reliability on HTTP uploads of 50 bytes
> at a time sounds to me a bit ... strained.  There *must* be simpler
> ways--and simpler goes a long way in the direction of trustworthy.