Question(s)

[avi.e.gross at gmail.com]

Agreed, Chris. There are many methods way better than the sort of RAID
architecture I supplied as AN EXAMPLE easy to understand. But even so, if a
hard disk or memory chip is fried or a nuclear bomb takes out all servers in
or near a city, you would need  some truly crazy architectures with info not
only distributed across the globe but perhaps also to various space
satellites or servers kept ever further out and eventually in hyperspace or
within a black hole (might be write-only, alas).

The point many of us keep saying is there can not easily or even with great
difficult, any perfect scheme that guarantees nothing will go wrong with the
software, hardware, the people using it and so on. And in the real world, as
compared to the reel world, many programs cannot remain static. Picture a
program that includes many tax laws and implementations that has to be
changed at least yearly as laws change. Some near-perfect code now has to
either be patched with lots of errors possible, or redesigned from scratch
and if it takes long enough, will come out after yet more changes and thus
be wrong.

A decent question you can ask is if the language this forum is supposed to
be on, is better in some ways to provide the kind of Teflon-coated code he
wants. Are there features better avoided? How do you make sure updates to
modules you use and trust are managed as they may break your code. Stuff
like that is not as abstract.

In my view, one consideration can be that when people can examine your
source code in the original language, that can open up ways others might
find ways to break it, more so than a compiled program that you only can
read in a more opaque way.


-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com at python.org> On
Behalf Of Chris Angelico via Python-list
Sent: Tuesday, October 24, 2023 9:41 PM
To: python-list at python.org
Subject: Re: Question(s)

On Wed, 25 Oct 2023 at 12:20, AVI GROSS via Python-list
<python-list at python.org> wrote:
> Consider an example of bit rot. I mean what if your CPU or hard disk has a
location where you can write a byte and read it back multiple times and
sometimes get the wrong result. To be really cautions, you might need your
software to write something in multiple locations and when it reads it back
in, check all of them and if most agree, ignore the one or two that don't
while blocking that memory area off and moving your data elsewhere. Or
consider a memory leak that happens rarely but if a program runs for years
or decades, may end up causing an unanticipated error.
>

True, but there are FAR more efficient ways to do error correction :)
Hamming codes give you single-bit correction and two-bit detection at
a cost of log N bits, which is incredibly cheap - even if you were to
go for a granularity of 64 bytes (one cache line in a modern Intel
CPU), you would need just 11 bits of Hamming code for every 512 bits
of data and you can guarantee to fix any single-bit error in any cache
line. The "if most agree, ignore the one or two that don't" design
implies that you're writing to an absolute minimum of three places,
and in order to be able to ignore two that disagree, you'd probably
need five copies of everything - that is to say, to store 512 bits of
data, you would need 2560 bits of storage. But with a Hamming code,
you need just 523 bits to store 512 reliably.

Here's a great run-down on how efficiently this can be done, and how
easily. https://www.youtube.com/watch?v=X8jsijhllIA

Side note: If we assume that random bit flips occur at a rate of one
every X storage bits, having redundant copies of data will increase
the chances of a failure happening. For example, using a naive and
horrendously wasteful "store 256 copies of everything" strategy, you
would be 256 times more likely to have a random bitflip, which is
insane :) You would also be able to guarantee detection of up to 128
random bitflips. But as you can see, this puts a maximum on your
storage ratio.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list