threads or queue for this task

[Bjorn Pettersen]

> From: James J. Besemer [mailto:jb at cascade-sys.com] 
> 
> >James J. Besemer wrote:
> >  
> >>Generally, Queues never get "full" and you generally don't care if 
> >>they're empty.
> >
> >You can define Queues that can hold a maximum number of items; [...]
> >
> >Setting size is just an optional embellishment, though sometimes it 
> >does pay dividends.
> >
> 
> Setting size has it's place but otherwise introduces unnecessary 
> complications.  Among other things, it forces the queue 
> writer to deal with the situation of what to do if the queue is full.
Signal error 
> back to its client?   Sleep and try again later?  Silently 
> discard the request?  Abort?
> 
> "Generally," NOT having a limit is by far the more elegant solution.

[snip]

You must be kidding (or you can't have written any significant amount of
multithreaded code). In the general producer/consumer paradigm you want
the producer to look very similar to:

  try:
      while 1:
          work = createWork()
          if not work: break
          queue.push(work) # block if queue full
      if I'm the last producer: # kill consumers
          for i in range(numberOfConsumerThreads):
              queue.push(None)
  except Timeout:
      abendGracefully()

and the consumer to look very similar to:

  try:
      while 1:
          work = queue.pop() # block until not empty
          if not work: break
          process(work)
  except Timeout:
      abendGracefully()

If you have any amount of data to process (why else would you be
multithreading?) I've never seen producers and consumers being close
enough in performance to not have the system end up swapping to death --
unless, that is, if the queue is bounded. The Python Queue module
supports most of this out-of-the-box, and adding timeouts is fairly
trivial...

processing-too-many-16Gb-files-with-tight-SLAs-leaves-scares<wink>'ly
y'rs
-- bjorn