Thread processing

[Paul Moore]

Dave Brueck <dave at pythonapocrypha.com> writes:

> And therein lies the problem, IMO. Again, maybe things would "feel" more
> right if you focus more on the work being done than on the threads. I
> mean, do you really care about which thread finishes next or is it the
> completion of a chunk of work?

Good point. I think part of the problem was that I'd sped my
processing up by something like 30 times, as a result of changing from
sequential processing to threads. So I was obsessed with the how
(speed everything up with threads) rather than the what.

> I know it's tempting to come up with a general-purpose framework, but to
> be honest, there's so little code involved that it might not be worth the
> effort - once you're familiar with the patterns then it takes about the
> same amount of effort to write it as to find/import/use a prebuilt
> framework.

You may be right. The problem I was having was that I'm not
particularly familiar with the patterns, so I was getting in a mess.
Each attempt worked (more or less) but what should have been minor
modifications were taking far too long. So I thought that if I got the
pattern sorted once and for all, that would solve my problem.

That'll teach me to go for a quick fix...

> Here's another pattern that's sometimes useful (coding from memory, not
> tested):

I know this as a thread pool. While it's short, I probably would
package it up in a module. In fact, I probably will :-)

> The above approach seems very natural to me because it is work-centric -
> you care about doing work and for all you know the worker pool could be a
> single thread, a fixed number of threads, no threads, whatever. Notice
> that the main thread doesn't ever explicitly wait for the threads to
> finish - they _do_ finish, of course, but the how and when isn't something
> the rest of the program really cares about.

You've hit on my problem, definitely! While I understand your logic
entirely, I'm twitching frantically, because I want to worry about
making sure there are enough threads, that I can check when the work's
finished, etc...

I don't need coding advice, I need a psychiatrist :-)

Seriously, thanks for helping me get to the bottom of the issue. I'd
basically got into too much of a knot with it, and needed to take a
step back. (Getting called off onto another job for most of today
helped as well, it gave me a breather from this one...)

> As someone else pointed out - your worker func could simply return a value
> (making it easy to test in isolation) and you could just wrap the worker
> function in some other function that takes the result and adds it to a
> work queue. This also generalizes well to the cases where there are no
> results to report - the worker function implicitly returns None anyway.

Yes.

Thanks for taking the time to help me with this.

Paul.
-- 
This signature intentionally left blank