[Fwd: Re: repeat something in a thread, but stop when the program stops]

[Steve Holden]

[Sorry about this: the non-appearance of this article might just be due 
to a glitch on my news server, but other articles posted since have made 
it. So if this appears two, or three, or four times please accept my 
apologies for the paranoia which made me send it via the mailing list].

-------- Original Message --------
Subject: Re: repeat something in a thread, but stop when the program stops
Date: Tue, 28 Sep 2004 20:45:00 -0400
From: Steve Holden <steve at holdenweb.com>
Newsgroups: comp.lang.python
References: <5db4fffa.0409270551.57a96cda at posting.google.com> 
<mailman.3969.1096301203.5135.python-list at python.org> 
<Xns95725B68F8800cpl19ghumspamgourmet at 195.20.224.116> 
<mailman.4020.1096389667.5135.python-list at python.org>

Tim Peters wrote:

> [Tim Peters, on waking up a thread to check a queue 20 times a second]
> 
>>>Measure it:  you won't be able to see a difference.  Modern processors
>>>cram on the order of 1E9 cycles into each second.  Doing a tiny amount
>>>of work 20 times per second is trivial.
> 
> 
> [Harald Massa]
> 
>>You are surely right, Tim. Measuring did not detect the difference. I just
>>see my tasktray collecting more and more icons of programs which just do a
>>very tiny amount of work every half a second, and ... as them getting many,
>>the computer begins to feel more sluggish.
> 
> 
> Those are quite different, and they do more work than you may think. 
> For example, if a program puts an icon in the Windows systray, then it
> has a graphical UI too, and has to run a Windows message pump to
> detect stuff like "is the mouse hovering over my icon now?  did they
> click on me?  did they right-click on me?  did they double-click on
> me?".  Apps in the systray often do a lot more than just that too. 
> For example, one app in my systray right now hooks itself into the
> Windows clipboard chain, and intercepts (and so also slows, by some
> amount I couldn't care less about) all clipboard operations.  Another
> hooks itself into the file system at a low level, tracking all disk
> changes, and so slows (by a measurable amount!) all disk writes. 
> Heck, the little systray icon showing the pressure on my laptop's
> touchpad represents a process that has read over a billion bytes since
> I last booted.  Etc.
> 
> A Python thread waking up to see whether a Python Queue has something
> in it does none of that stuff.  If you have hundreds of Python threads
> doing this, then you can start to think about maybe worrying when it
> gets to thousands <wink>.

[I'm beginning to get seriously worried about Mozilla, as two replies
have now disappeared into the ether with no trace, even in Sent"!].

AS a data point, I have just finished my first serious use of threads,
in a program that email monthly statements. With 14,000+ statements each
month and a 20-second socket timeout on non-existent and slow SMTP
servers, the run length of the single-threaded solution was getting way
too long.

So I recast the program as a manager main thread that generates a number
of worker threads, each with its own input Queue. The manager sends
database rows to each of the worker threads (double-buffering, though I
suspect that isn't necessary) and the workers try to send the
appropriate email and return a result on a common unlimited Queue.

In testing, with fifty threads active the CPU utilization was reading
2-3% for most of the time, emphasizing that much of network
communication is waiting for things to happen. I'm thinking about
increasing the number of threads still further for production, but I
have little experience to guide me, so advice will be welcomed.

A further benefit of the exercise is that the program as refactored is
much cleaner, easier to understand and better coupled than it previously
was. In case it's of interest I attach (to avoid word-wrap) the code of
the proof-of-concept demonstrator I hacked together before starting work
on the production program.

regards
   Steve
-- 
XXX Please note recent change of email address
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thrtest.py
Type: application/x-python
Size: 2521 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20040928/20300d84/attachment.bin>