queue deadlock possibility

[Michael Bayer]

Hi -

i was just going through this thread: http://mail.python.org/ 
pipermail/python-list/2006-April/336948.html , where it is suggested  
that the Lock instance used by Queue.Queue should be publically  
configurable.  I have identified another situation where a Queue can  
be deadlocked, one which is also alleviated by configuring the type  
of Lock used by the Queue (or just changing it to an RLock).

The scenario arises when the Queue is operated upon within the  
__del__ method of an object; since __del__ can be called at somewhat  
unpredictable times, I have observed that it is in fact possible, in  
extremely rare circumstances, for put() to be called within a get()  
or possibly vice versa; since both methods lock on the same  
underlying mutex object which is an instance of threading.Lock, a  
deadlock occurs.

The issue can be fixed by substituting a threading.RLock for the  
threading.Lock object that Queue instantiates by default.

The scenario this has arisen within is a database connection pool,  
which puts connections in a Queue, returns them via get() within a  
wrapper object, and the wrapper object automatically returns the  
connection to the Queue via put() within its __del__ method (an  
explicit close() method is available as well).  While I cant  
reproduce it locally, one of my users experiences it regularly.  I  
had him install the "threadframe" module to trace it out, and it  
reveals that all threads are hung within Queue on the acquiring of  
the "not_empty" and "not_full" Conditionals, and the offending stack  
trace within it looks like this:

   File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 84, in  
connect
   File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 130, in  
__init__
   File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 102, in  
get
   File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 226, in  
do_get
   File "/usr/lib/python2.4/Queue.py", line 116, in get
     raise Empty
   File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 157, in  
__del__
   File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 163, in  
_close
   File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 99, in  
return_conn
   File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 216, in  
do_return_conn
   File "/usr/lib/python2.4/Queue.py", line 71, in put
     self.not_full.acquire()

this is a simplified version of the logic, the actual version is the  
pool.py module in the SQLAlchemy package:

import Queue

pool = Queue.Queue(maxsize=10)

class ConnectionWrapper(object):
     def __init__(self, connection):
         self.connection = connection
     def __del__(self):
         pool.put_nowait(self)

# fill up the pool with 10 connections
for x in range(0,10):
     pool.put_nowait(database.connect())

def connect():
     return ConnectionWrapper(pool.get())

At the moment I am modifying the Queue's mutex to be a  
threading.RLock to fix the problem;  what does the community think of  
either making the Queue's Lock instance public or changing it to an  
RLock ?

- mike