Multithreading and Queue

[robert]

Jeffrey Barish wrote:
> Several methods in Queue.Queue have warnings in their doc strings that they
> are not reliable (e.g., qsize).  I note that the code in all these methods
> is bracketed with lock acquire/release.  These locks are intended to
> protect the enclosed code from collisions with other threads.  I am
> wondering whether I understand correctly that the reason these methods are
> still not reliable is that from the point where a thread calls qsize (for
> example) to the point in Queue where a thread acquires a lock there is a
> bunch of code, none of which is protected by a lock, (and moreover there is
> another bunch of code between the point where a thread releases a lock and
> then actually returns to the calling program) and so despite the locks in
> Queue it is still possible for values to change before a thread acts on
> them.

That warnings are quite meaningless: If you get a certain q size, it 
might be changed the next moment/OP anyway.

simply len() is the same as that

     def qsize(self):
         """Return the approximate size of the queue (not reliable!)."""
         self.mutex.acquire()
         n = self._qsize()
         self.mutex.release()
         return n
     ...
     def _qsize(self):
         return len(self.queue)


All that time-consuming locking in Queue is quite unnecessary. As 
assumed in many other locations in the std lib, list.append() / .pop() / 
len() are atomic. Thus doing in a single location a IndexError-catch on 
a .pop() race, and all care is done.

(the only additional guarantee through that lock's ( at .append() time) 
is that the q size is never one over the exact "maximum" - but thats 
pedantic in a threaded multi-producer mess.)

The whole CallQueue in 
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/491281 for 
example doesn't use a single lock - though its an inter-thread 
communication hot spot.

-robert