file locking...

[bruce]

hi nigel...

using any kind of file locking process requires that i essentially have a
gatekeeper, allowing a single process to enter, access the files at a
time...

i can easily setup a file read/write lock process where a client app
gets/locks a file, and then copies/moves the required files from the initial
dir to a tmp dir. after the move/copy, the lock is released, and the client
can go ahead and do whatever with the files in the tmp dir.. thie process
allows multiple clients to operate in a psuedo parallel manner...

i'm trying to figure out if there's a much better/faster approach that might
be available.. which is where the academic/research issue was raised..

the issue that i'm looking at is analogous to a FIFO, where i have lots of
files being shoved in a dir from different processes.. on the other end, i
want to allow mutiple client processes to access unique groups of these
files as fast as possible.. access being fetch/gather/process/delete the
files. each file is only handled by a single client process.

thanks..



-----Original Message-----
From: python-list-bounces+bedouglas=earthlink.net at python.org
[mailto:python-list-bounces+bedouglas=earthlink.net at python.org]On Behalf
Of Nigel Rantor
Sent: Sunday, March 01, 2009 2:00 AM
To: koranthala
Cc: python-list at python.org
Subject: Re: file locking...


koranthala wrote:
> On Mar 1, 2:28 pm, Nigel Rantor <wig... at wiggly.org> wrote:
>> bruce wrote:
>>> Hi.
>>> Got a bit of a question/issue that I'm trying to resolve. I'm asking
>>> this of a few groups so bear with me.
>>> I'm considering a situation where I have multiple processes running,
>>> and each process is going to access a number of files in a dir. Each
>>> process accesses a unique group of files, and then writes the group
>>> of files to another dir. I can easily handle this by using a form of
>>> locking, where I have the processes lock/read a file and only access
>>> the group of files in the dir based on the  open/free status of the
>>> lockfile.
>>> However, the issue with the approach is that it's somewhat
>>> synchronous. I'm looking for something that might be more
>>> asynchronous/parallel, in that I'd like to have multiple processes
>>> each access a unique group of files from the given dir as fast as
>>> possible.
>> I don't see how this is synchronous if you have a lock per file. Perhaps
>> you've missed something out of your description of your problem.
>>
>>> So.. Any thoughts/pointers/comments would be greatly appreciated. Any
>>>  pointers to academic research, etc.. would be useful.
>> I'm not sure you need academic papers here.
>>
>> One trivial solution to this problem is to have a single process
>> determine the complete set of files that require processing then fork
>> off children, each with a different set of files to process.
>>
>> The parent then just waits for them to finish and does any
>> post-processing required.
>>
>> A more concrete problem statement may of course change the solution...
>>
>>    n
>
> Using twisted might also be helpful.
> Then you can avoid the problems associated with threading too.

No one mentioned threads.

I can't see how Twisted in this instance isn't like using a sledgehammer
to crack a nut.

   n
--
http://mail.python.org/mailman/listinfo/python-list