analyzing lock contention in multi-threaded app?

[Skip Montanaro]

I was reading the PEP 319 thread on python-dev yesterday and a question
popped into my brain with recurs (to me) from time-to-time.  Is there a
decent way to analyze a multi-threaded app for lock contention?  Let me make
this more concrete.  I have an XML-RPC server which contains a handful of
threading.RLock objects to protect shared data and one Queue.Queue object
with a max size of 5 which maintains a pool of database connections.  All my
acquire() and get() calls are of the blocking variety.  Is there some way to
reasonably monitor lock contention?  I currently have a method which returns
lock status:

    def get_lock_info(self):
        return {
            'query_lock': self.query_lock._RLock__block.locked(),
            'sql_lock': self.sql_lock._RLock__block.locked(),
            'namemap_lock': self.namemap_lock._RLock__block.locked(),
            'top_50_lock': self.top_50_lock._RLock__block.locked(),
            'log_lock': self.log_lock._RLock__block.locked(),
            'conn_pool': self.conn_pool.empty(),
            'dump_lock': self.dump_lock._RLock__block.locked(),
            }

Since it doesn't lock anything itself, it is guaranteed to not block.  (Of
course, it might occasionally return incorrect data as well, but I'm hoping
to examine trends.)  I have a little script which executes the above call
periodically and tells me if a lock is locked or the queue is empty.  While
useful to a certain degree, it doesn't tell me if any other acquire() or
get() calls have blocked.  What other techniques are available to monitor
the lock contention of their multi-threaded apps?

Thx,

Skip