Threaded Design Question

[half.italian at gmail.com]

On Aug 9, 12:09 pm, "Justin T." <jmtull... at gmail.com> wrote:
> On Aug 9, 11:25 am, half.ital... at gmail.com wrote:
>
> > Here's how I have it designed so far.  The main thread starts a
> > Watch(threading.Thread) class that loops and searches a directory for
> > files.  It has been passed a Queue.Queue() object (watch_queue), and
> > as it finds new files in the watch folder, it adds the file name to
> > the queue.
>
> > The main thread then grabs an item off the watch_queue, and kicks off
> > processing on that file using another class Worker(threading.thread).
>
> Sounds good.
>
>
>
> > I made definite progress by creating two queues...watch_queue and
> > processing_queue, and then used lists within the classes to store the
> > state of which files are processing/watched.
>
> This sounds ugly, synchronization is one of those evils of
> multithreaded programming that should be avoided if possible. I see a
> couple of dirt simple solutions:
>
> 1. Have the watch thread move the file into a "Processing" folder that
> it doesn't scan
> 2. Have the watch thread copy the file into a python tempfile object
> and push that onto the queue, then delete the real file. This can be
> done efficiently (well, more efficiently than new.write(old.read())
> with shutil.copyfileobj(old, new)
>
> Both those take very few lines of code, don't require synchronization,
> and don't require extending standard classes.

Thanks foor the ideas Justin.

I started subclassing/extending the Queue.Queue object with two
additional classes.  One to return the first item in the list without
removing it from the queue, and one to return all items of the list
without removing them.  I think this will take me to where I want to
go.  If it doesn't work, I might just use your processing folder
approach.  That sounds the easiest, although I'm still interested in
any idioms or other proven approaches for this sort of thing.

~Sean