file locking...

[Thomas Guettler]

Hi Bruce,

you can do it like Maildir [1] you move (os.rename()) file or directories.

Maybe something like this: You have three directories: "todo", "in-process" and "done".
A process tries to os.rename from todo to in-process. If it fails, some other
process has done it before. If the process is done it moves the file/directory
to "done".

To avoid stressing the directories, too much, It might be good to use subdirectories
like todo/NN/MM/. I think git (version control system created by Linus Torvalds)
does something like this.


  Thomas

[1] http://wiki.dovecot.org/MailboxFormat/Maildir
    This page describes Maildir and some unneeded parts of the specification.

bruce schrieb:
> Hi.
> 
> Got a bit of a question/issue that I'm trying to resolve. I'm asking this of
> a few groups so bear with me.
> 
> I'm considering a situation where I have multiple processes running, and
> each process is going to access a number of files in a dir. Each process
> accesses a unique group of files, and then writes the group of files to
> another dir. I can easily handle this by using a form of locking, where I
> have the processes lock/read a file and only access the group of files in
> the dir based on the  open/free status of the lockfile.
> 
> However, the issue with the approach is that it's somewhat synchronous. I'm
> looking for something that might be more asynchronous/parallel, in that I'd
> like to have multiple processes each access a unique group of files from the
> given dir as fast as possible.
> 
> So.. Any thoughts/pointers/comments would be greatly appreciated. Any
> pointers to academic research, etc.. would be useful.
> 
> thanks
> 
> 
> 


-- 
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de