Debugging SocketServer.ThreadingTCPServer

[R. Bernstein]

"Stuart D. Gathman" <stuart at bmsi.com> writes:

> On Tue, 16 Jan 2007 09:11:38 -0500, Jean-Paul Calderone wrote:
> 
> > On Tue, 16 Jan 2007 00:23:35 -0500, "Stuart D. Gathman"
> > <stuart at bmsi.com> wrote:
> >>I have a ThreadingTCPServer application (pygossip, part of
> >>http://sourceforge.net/projects/pymilter).  It mostly runs well, but
> >>occasionally goes into a loop.  How can I get a stack trace of running
> >>threads to figure out where the loop is?  Is there some equivalent of
> >>sending SIGQUIT to Java to get a thread dump?  If needed, I can import
> >>pdb and set options at startup, but there needs to be some external way
> >>of triggering the dump since I can't reproduce it at will.

The problem here is that signals are handled only in the main
thread. If that thread is blocked for some reason, your signals will
also be blocked. 

Given that you were considering using pdb let me suggest instead pydb
- http://bashdb.sf.net/pydb; pdb has no notion of threads but pydb
does if you give it the --threading option. (It's thread support might
stand a bit of improvement, but at least it's there and is as good or
better than any other Python debugger that I am aware of.)

> > 
> > Grab the gdbinit out of Python SVN Misc/ directory.  Apply this patch:
> > 
> >   http://jcalderone.livejournal.com/28224.html
> > 
> > Attach to the process using gdb.  Make sure you have debugging symbols
> > in your build of Python.  Run 'thread apply all pystack'.
> 
> Did this.  gdb displays main thread fine (waiting on accept(), duh).  But
> gdb goes into a loop displaying the first worker thread.  There are no
> extension modules other than the batteries included ones. In this
> application, I believe, only _socket.  (I.e. a pure python server.)
> 
> I will try for a C stack trace next time it loops.
> 
> Also, the looping server needs kill -9.  SIGTERM and SIGINT won't stop it.
> And after it dies with SIGKILL, the port is still in use for 5 minutes or
> so (and the server cannot be restarted).

See again above why SIGTERM and SIGINT might not necessarily do
anything. But if you get into pydb with thread support, at least you
should be able to determine if you are blocked in the main thread; so
you can know if a SIGTERM or SIGINT will be seen.

And in pydb one can kill specific threads, but you do this at
your own risk because you could cause a deadlock here. As a
convenience, there is a debugger "kill" command so you don't have to
remember the pid. And that's there because it is really the only
reliable way I know how to terminate the program. There is also a
"signal" debugger command if you wanted to issue SIGTERM or SIGINT
signals. 

> 
> This is really making me appreciate Java.

Note that Java has preemptive threads, Python does not.