Queue.get_nowait() sometimes bogusly returns Empty

[Tim Peters]

[Geoffrey Talvola]
> ...
> I understand what you're saying.  Would it make a difference if I told
> you that for this application, it's OK if the "get_nowait_reliable"
> operation blocks for "a little while"?  In other words, I don't mind if
> it blocks while 17 other threads manipulate the queue.  What I care
> about is that once the queue is no longer being manipulated in other
> threads, I either get an Empty exception or an object out of the queue.
> For my purposes if the queue was empty at any point during the operation
> then raising the Empty exception is OK.
>
> My requirements are a bit fuzzier than the strict no-blocking
> requirement of Queue.get_nowait() so maybe it's possible to implement
> this.
>
> In case you're wondering, this is in Webware's WebKit application server
> where there is a pool of N threads, and a queue of (non-thread-safe)
> servlet instances acting as a cache.  We only want to create servlet
> instances when all existing instances are in-use.  When a given thread
> needs a servlet, it basically does:
>
> 	try:
> 		instance = cache.get_nowait()
> 	except Queue.Empty:
> 		instance = new_instance()
> 	# use the instance...
> 	cache.put(instance)
>
> The problem is that every once in a while get_nowait() returns Empty
> when the queue isn't actually empty, so the size of the cache grows
> slowly and unboundedly over time.

How do you know that the unbounded growth is *due* to that get_nowait()
returns Empty at odd times?  If service requests follow the usual sorts of
lumpy distributions, the code you showed above simply will create an
unbounded number of instances over time, since it caters to peak
simultaneous usage as if resources were unbounded.  Fiddling the Queue
implementation isn't going to cure that, if so.

> But I'd prefer for the cache to contain at most N instances where N is
> the number of threads.

In that case, why not use a bounded Queue (the Queue constructor has an
argument for this purpose) of maximum size N, and use blocking get?  That
gives up the illusion that resources are unbounded, but the heart of your
complaint is that writing code as if resources were unbounded leads to
unbounded resource usage <wink>.  If your threads are returning instances
correctly to the queue, then the block will never wait long -- unless it's
really the case that N instances aren't actually enough to service the
threads.