Queue.get_nowait() sometimes bogusly returns Empty

[Geoffrey Talvola]

aahz at pythoncraft.com wrote:
> In article <mailman.1026837865.32291.python-list at python.org>,
> Geoffrey Talvola  <gtalvola at nameconnector.com> wrote:
> >
> > In case you're wondering, this is in Webware's WebKit 
> > application server
> > where there is a pool of N threads, and a queue of 
> > (non-thread-safe) servlet
> > instances acting as a cache.  We only want to create servlet 
> > instances when
> > all existing instances are in-use.  When a given thread 
> > needs a servlet, it
> > basically does:
> >
> >	try:
> >		instance = cache.get_nowait()
> >	except Queue.Empty:
> >		instance = new_instance()
> >	# use the instance...
> >	cache.put(instance)
> >
> >The problem is that every once in a while get_nowait() 
> > returns Empty when
> > the queue isn't actually empty, so the size of the cache 
> > grows slowly and
> > unboundedly over time.  But I'd prefer for the cache to 
> > contain at most N
> > instances where N is the number of threads.
> 
> You've got several options.  Simplest is to change the 
> exception so that
> it only creates new instances when there are fewer than N instances.  
> 
> What I'd consider doing, though, is make a data queue instead of a
> servlet cache queue, such that all the servlet threads grab their data
> from that data queue.  Then you have a separate monitor thread that
> periodically checks the data queue and creates new servlet 
> threads when
> it hits "too many" items.  Conversely, if the data queue stays empty,
> you delete servlet instances (by passing quit tokens in the data queue
> would be simplest).

There is something like that already in WebKit that handles the overall
request processing.

What I'm talking about here is essentially caches of individual servlets,
where loosely speaking a servlet corresponds to a unique URI.  You may have
10 threads, but 2000 different servlet classes in your application.  Each of
those servlet classes has a cache with anywhere from 0 through 10 instances,
depending on the maximum number of simultaneous accesses of that servlet.
We only want to allocate a new instance when all existing instances of that
servlet class are busy.  That's what the catching of Queue.Empty was
intended to detect, but it in fact doesn't work.

As mentioned in a response to Tim, I switched to using a list with .append()
and .pop() and it is working fine.  It actually functions as a stack instead
of a queue but that's fine with me.

- Geoff