Unnoticed traceback in a thread

[Skip Montanaro]

I have a program which is almost always running in a single thread. It
uses a daemon thread (via threading.Timer) to periodically checkpoint
some state. The program runs for days at a time.

Over the past couple days, two instances of the subthread croaked with
tracebacks because while they were iterating over the checkpointable
data, the main thread modified the data. Blammo! It was an easy fix
(threading.Lock) and everything is back on an even keel.

After the bug was tickled, the program continued to do its thing. It
just stopped checkpointing. Given the way the program is managed, the
traceback wound up in an obscure log file, and it was a couple days
before I noticed it. I've seen this sort of
thread-dies-but-program-doesn't-crash situation before. I can look at
ways to more actively monitor the contents of the obscure log file,
but is there some non-hackish way that the demise of the daemon thread
can take down the entire program? (Sometimes it's good that it doesn't
crash. Other times it would at least be handy if it did.)

Thx,

Skip