Code For Five Threads To Process Multiple Files?

[A.T.Hofkamp]

On 2008-05-21, tdahsu at gmail.com <tdahsu at gmail.com> wrote:
> I'd appreciate any help.  I've got a list of files in a directory, and
> I'd like to iterate through that list and process each one.  Rather
> than do that serially, I was thinking I should start five threads and
> process five files at a time.
>
> Is this a good idea?  I picked the number five at random... I was

Depends what you are doing.
If you are mainly reading/writing files, there is not much to gain, since 1
process will already push the disk IO system to its limit. If you do a lot of
processing, then more threads than the number of processors is not much use. If
you have more 'burtsy' behavior (first do lot of reading, then lot of
processing, then again reading, etc), then the system may be able to do some
scheduling and keep both the processors and the file system busy.

I cannot really give you advice on threading, I have never done that. You may
want to consider an alternative, namely multi-tasking at OS level. If you can
easily split the files over a number of OS processes (written in Python), you
can make the Python program really simple, and let the OS handle the
task-switching between the programs.

Sincerely,
Albert