Thread limits?

[Aahz Maruch]

In article <8np2s8$1jbm$1 at newssvr05-en0.news.prodigy.com>,
Quasimodo <KILLzdramboSPAMMERS at zdnetmail.com> wrote:
>
>Thanks for the input.  I'm aware each PC is going to have a small limit and
>to resolve that  I'm planning on distributing the load on mulitple machines.

Or use a multi-processor machine.

>If I understand your suggestion correctly, you're suggesting spawning some
>threads (10, say) then waiting for them to complete their tasks, thereby
>getting destroyed, then spawning more from the Queue as current ones are
>destroyed?

Actually, the spawn time for threads is significant.  More efficient to
keep that thread pool active and have each thread just pick up new work
as it finishes old work.

>Sounds reasonable - I'll have to try it.  My only concern is I'm trying to
>get this as close to realtime monitoring as possible (huge overhead so
>settled on per minute scans).  I can't trigger a change event since the
>source updates in realtime (100's of times per minute).  Also I need to scan
>approx 5,000 different items each with it's own potential source.  So I
>might get into a situation where, if each thread takes 10 seconds to
>process, I'll only get 60 processed per minute where the max requirement is
>5,000 per minute (practical max probably around 1,000).
>
>I'm guessing this is going to be much along the lines of trial and error
>combining both elements (i.e. running as many threads as possible, being fed
>from a queue).

Real-time work is scary.  You may well be better off spawning objects in
some kind of server that receives update requests, then having the
server associate threads with objects that need updating.  For
efficiency, you may want to update each individual object only once per
X minutes, batching updates per object.

WARNING: Python has something called the Global Interpreter Lock (GIL).
This means that only one thread can perform real Python code at any
time, no matter how many CPUs you have (this is a per-process
limitation).  The only way to work around this limitation is to call C
code that releases the GIL.  All the blocking I/O calls in Python do
this automatically, so threading is an easy to boost I/O throughput.
--
                      --- Aahz (Copyright 2000 by aahz at pobox.com)

Androgynous poly kinky vanilla queer het    <*>     http://www.rahul.net/aahz/
Hugs and backrubs -- I break Rule 6

"A foolish consistency is the hobgoblin of little minds, adored by little 
statesmen and philosophers and divines."  -- Ralph Waldo Emerson