Threaded Design Question

[Justin T.]

On Aug 9, 11:25 am, half.ital... at gmail.com wrote:
>
> Here's how I have it designed so far.  The main thread starts a
> Watch(threading.Thread) class that loops and searches a directory for
> files.  It has been passed a Queue.Queue() object (watch_queue), and
> as it finds new files in the watch folder, it adds the file name to
> the queue.
>
> The main thread then grabs an item off the watch_queue, and kicks off
> processing on that file using another class Worker(threading.thread).
>
Sounds good.

>
> I made definite progress by creating two queues...watch_queue and
> processing_queue, and then used lists within the classes to store the
> state of which files are processing/watched.
>
This sounds ugly, synchronization is one of those evils of
multithreaded programming that should be avoided if possible. I see a
couple of dirt simple solutions:

1. Have the watch thread move the file into a "Processing" folder that
it doesn't scan
2. Have the watch thread copy the file into a python tempfile object
and push that onto the queue, then delete the real file. This can be
done efficiently (well, more efficiently than new.write(old.read())
with shutil.copyfileobj(old, new)

Both those take very few lines of code, don't require synchronization,
and don't require extending standard classes.