Drop folder and race conditions

[Steven D'Aprano]

On Tue, 09 Oct 2007 07:05:57 -0500, Larry Bates wrote:

> I have a need to implement a drop folder upload mechanism for secure
> uploading of files to a server.  At first glance this appears that it
> would be an easy application to write.  Then I begin to think about the
> race conditions that exist between the process that will wake up to
> upload the files and the fact that the user can add additional files to
> the drop folder at any point in time. I would like to clear out
> files/folders after they have been processed but it is possible that
> while the background process was uploading files, the user added
> additional files/folders to the drop folder that make the deletion of
> folders impossible.  This project has become significantly more complex
> than it appeared at first.  Anyone out there have any "sage" advice on
> how to tackle this beast?


Off the top of my head...

Recursively copy the contents of each folder to the server, deepest 
first, deleting each file as it's copied. Then delete the folder as soon 
as you've emptied it (but naturally not the top-level drop folder). If it 
happens to *not* be empty (because the user has added additional files to 
it), just catch the error and ignore it. In another few minutes, you'll 
try again, copying the newly added files and then delete the folder.

Because each file is being deleted as soon as it is copied, you'll won't 
build up an ever-increasing collection of files. At worst, you may have a 
collection of folders -- but unless you're fighting a hostile process 
that can add folders quicker than you can delete them, you'll eventually 
win.

Maybe you need a heuristic that says "if the number of 'directory not 
empty' errors keeps increasing, assume we're dealing with a hostile 
process and take extra steps". Say, lock the drop folder for five minutes 
while you flush it.

Another possibility is that before you start copying from the folders, 
you change their permissions to prohibit the user adding extra files into 
them. That way, nothing can be added to them once you start copying, and 
you can be sure that you can delete them once empty.


-- 
Steven