Python from Wise Guy's Viewpoint

[Steve Schafer]

On 20 Oct 2003 22:08:30 +0200, Pascal Bourguignon
<spam at thalassa.informatimago.com> wrote:

>AFAIK, while this parameter was out of range, there was no instability
>and the rocket was not uncontrolable.

That's perfectly true, but also perfectly irrelevant. When your
carefully designed software has just told you that your rocket, which,
you may recall, is traveling at several thousand metres per second, has
just entered a "can't possibly happen" state, you don't exactly have a
lot of time in which to analyze all of the conflicting information and
decide which to trust and which not to trust. Whether that sort of
decision-making is done by engineers on the ground or by human pilots or
by some as yet undesigned intelligent flight control system, the answer
is the same: Do the safe thing first, and then try to figure out what
happened.

All well-posed problems have boundary conditions, and the solutions to
those problems are bounded as well. No matter what the problem or its
means of solution, a boundary is there, and if you somehow cross that
boundary, you're toast. In particular, the difficulty with AI systems is
that while they can certainly enlarge the boundary, they also tend to
make it fuzzier and less predictable, which means that testing becomes
much less reliable. There are numerous examples where human operators
have done the "sensible" thing, with catastrophic consequences.

>My point.

Well, actually, no. I assure you that my point is very different from
yours.

>This "can't possibly happen" failure did happen, so clearly it was not
>a "can't  possibly happen" physically, which means that the problem was 
>with the software.

No, it still was a "can't possibly happen" scenario, from the point of
view of the designed solution. And there was nothing wrong with the
software. The difficulty arose because the solution for one problem was
applied to a different problem (i.e., the boundary was crossed).

>it would  be better to have smarter software that can try and handle
>UNEXPECTED error situations

I think you're failing to grasp the enormity of the concept of "can't
possibly happen." There's a big difference between merely "unexpected"
and "can't possibly happen." "Unexpected" most often means that you
haven't sufficiently analyzed the situation. "Can't possibly happen," on
the other hand, means that you've analyzed the situation and determined
that the scenario is outside the realm of physical or logical
possibility. There is simply no meaningful means of recovery from a
"can't possibly happen" scenario. No matter how smart your software is,
there will be "can't possibly happen" scenarios outside the boundary,
and your software is going to have to shut down.

>I would feel safer in an AI rocket.

What frightens me most is that I know that there are engineers working
on safety-critical systems that feel the same way. By all means, make
your flight control system as sophisticated and intelligent as you want,
but don't forget to include a simple, reliable, dumber-than-dirt
ejection system that "can't possibly fail" when the "can't possibly
happen" scenario happens.

Let me try to summarize the philosophical differences here: First of
all, I wholeheartedly agree that a more sophisticated software system
_may_ have prevented the destruction of the rocket. Even so, I think the
likelihood of that is rather small. (For some insight into why I think
so, you might want to take a look at Henry Petroski's _To Engineer is
Human_.) Where we differ is how much impact we believe that more
sophisticated software would have on the problem. I get the impression
that you believe that an AI-based system would drastically reduce
(perhaps even eliminate?) the "can't possibly happen" scenario. I, on
the other hand, believe that even the most sophisticated system enlarges
the boundary of the solution space by only a very small amount--the area
occupied by "can't possibly happen" scenarios remains far greater than
that occupied by "software works correctly and saves the rocket"
scenarios.

-Steve