analyzing lock contention in multi-threaded app?

[Tim Peters]

[Skip]
> I was reading the PEP 319 thread on python-dev yesterday and a
> question popped into my brain with recurs (to me) from time-to-time.
> Is there a decent way to analyze a multi-threaded app for lock
> contention?

No, locks are too low-level.  If you have to use locks, ensure that no
thread ever requires holding more than one lock at a time.  Then deadlock is
impossible.  If a thread has to hold more than one lock at a time, then
ensure that all threads acquire locks in a fixed global order; then deadlock
is also impossible.  Regardless of approach, strive to hold locks over as
small a stretch of code as possible.

Many uses of locks would be better served by using condition variables
instead, which are much better behaved than locks.  For example, here's the
canonical deadlock scenario:

def deadlock(obj1, obj2):
    obj1.acquire()
    obj2.acquire()

If A and B are locks, and thread1 calls deadlock(A, B) while thread2 calls
deadlock(B, A), deadlock is likely.  "Fixed global order" can be tricky to
ensure when locks are passed as arguments.

The condvar protocol is:

    # Somebody waiting for something to be true.
    cond.acquire()
    while what we're waiting for isn't true:
        cond.wait()   # this releases cond too
    # What we're waiting for is true now, and cond is acquired.
    ... do something ...
    cond.notify()
    cond.release()


    # Somebody else mutating the values that go into deciding whether
    # what we're waiting for is true.
    cond.acquire()
    ... mutate the values ...
    cond.notify()
    cond.release()

This shifts the focus from trying to outguess concurrent control flow, to
concentrating on the *logic* of "what we're waiting for".  This can be
valuable even in the "deadlock" example:

def nodeadlock(obj1, obj2):
    obj1.acquire()
    while not obj2.acquire(0):
        obj1.wait()
    # Now we hold obj1 and obj2.
    ... do something ...
    obj1.notify()
    obj1.release()
    obj2.notify()
    obj2.release()

If A and B are condvars, and thread1 calls nodeadlock(A, B) while thread2
calls nodeadlock(B, A), no problem.  The condvar protocol ensures that
nobody holds the condvar lock for a long time.  Sooner or later (and usually
sooner), one of the threads will acquire both the condvars.  Note that the
while-loop is a natural place to put logging info, if you're interesting in
finding out how often contention occurs.