Threaded Design Question

[Graeme Glass]

Using IPC is just adding needles complexity to your program. Instead
of constantly scanning the directory for files and then adding them to
a Queue, and then having to worry if that specific file may have
already been popped off the queue and is currently running by one of
the workers, just poll the directory for a specific event, ala
CREATE.

If you are using a *nix based system, you can use, pyinotify (http://
pyinotify.sourceforge.net/) using EventsCodes.IN_CREATE event.
Else if you using a windows system, take a look at:
http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_changes.html,
Tim has a couple of different ideas, sure one of them will be what you
need.

Then all you have to do is added newly created files to the queue,
fork the off to workers and the workers can delete them when they
done. No need to worry if someone is working with it already, as it
will only be added once to the queue, when it is created. HTH.

Regards,
Graeme