poking around a running program

[Paul Rubin]

I have a long-running program that has lots of net connections open on
separate threads.  I'm twiddling my thumbs waiting for it to finish
(each run takes maybe an hour) and although I'm logging various info
that I can monitor as the run progresses, it would be cool to be able
to actually poke around the program's data while it's running.  

I'm thinking of adding a "console" thread that would basically be a
read-eval-print loop that I could use to inspect data interactively.
Maybe it would just eval stuff or maybe it would use pdb.  Of course
it would be a bit dangerous since I could mutate things with it but I
guess it would be enough to just be careful not to do that.

Is this kind of thing common?  Has it been done before and are there
some recipes?  I did a quick search and didn't find anything obvious.
One thing I'm wondering is how to examine the internal state of
iterators, especially the C implementations from itertools.

More advanced might be some kind of hot-patching scheme (to be able to
change the code without stopping it) but for now I think I'm ok just
restarting the program every so often, modifying the code between runs
based on what happens in a given run.