Threading help?

[Cliff Wells]

On Wed, 06 Mar 2002 20:07:59 -0500
Peter Hansen wrote:

> Cliff Wells wrote:
> > 
> > On Wed, 6 Mar 2002 13:26:12 -0800
> > Cliff Wells wrote:
> > 
> > > The approach I would take, based upon the information given, is to
have
> > > thread A retrieve the data every .1s (using time.sleep),  when the
data
> > is
> > > retrieved, put it on a Queue and go back to sleep.  Thread B blocks
on
> > the
> > > Queue until data is available.  When data becomes available, B
processes
> > > that data and place it on a second Queue for thread C (thread C
handling
> > > HTTP requests) to deal with.
> > 
> > Hm.  Okay, I had to reconsider this.  Clearly if the processing is
slower
> > than .1s and data is being added to it every .1s, the Queue is going to
> > endlessly grow as more data is added to it.  If this is the case, it
might
> > make sense to have more than one consumer thread (B) doing the
processing.
> 
> I might be missing something, but I don't see how adding another thread
> (with a slight extra overhead associated with it) would actually increase
> performance for what I infer is CPU-bound processing in thread B.  
> Threads don't add performance capacity, they remove it!  (Or is B
blocking
> on something like a socket, thus slowing it down?)

Yes, I thought of this, but I believe that this would give the net effect
of increasing thread priority for the B threads (since they will get more
CPU time as a whole vs threads A and C). 

But maybe a better approach would be this: if processing time is greater
than .1s and we don't care about out-of-date data (big assumption, but not
unreasonable), then use the Queue between A and B and simply have B empty
the Queue on every iteration, processing only the newest data.  This would
keep the Queue to a reasonable size at the cost of dropping data.



-- 
Cliff Wells, Software Engineer
Logiplex Corporation (www.logiplex.net)
(503) 978-6726 x308  (800) 735-0555 x308