multiple instance on Unix

[C Ginger]

I know the approach to creating a lock file has been around a long time 
but there are certain weaknesses to it. There are a number of race 
conditions in it. For instance if process A detects the directory isn't 
there it will attempt to create it. During that same time process B 
might also not find it there - since A hasn't completed its create yet.

I think a better approach would be to use the shared memory option (see 
man shmop). The idea here is that each invoked instance would get back 
the same shared memory and use an appropriate atomic op to lock the 
process. Of course doing this in pure Python might be a bit difficult.

Chuck Wegrzyn


Jeremy Jones wrote:
> Nigel King wrote:
> 
>> Jeremy,
>> I have not explained very well what I wanted.
>>
>> I had a program that was called randomly by specific emails arriving 
>> which asked for certain information. If two or more emails arrived 
>> simultaneously then procmail asked two or more instances of my program 
>> to run. These instances interfered with one another, so I needed a 
>> process to stop that from happening. What my son devised was for the 
>> first to create a directory and run and when finished to delete the 
>> directory. The subsequent instances could try to create a directory 
>> but fail in an atomic piece of code. They would sleep for 1 or more 
>> seconds and then try again. The first of the subsequent instances that 
>> tries and succeeds stops the others from succeeding.
>>
>> Now, this works but I wondered whether anybody knew of a more standard 
>> bit of python code was available for ensuring that only one instance 
>> was processing. mutex does it for threads but not for instances as I 
>> understand it.
>>
>> The specification for a better process would include the ability to 
>> ensure that the queue was orderly, in other words some sort of FIFO 
>> would ensure that first served would have been the first to request 
>> the lock and fail.
>>
>> Our solution which does not satisfy the previous paragraph.
>>
>> import os, time
>> try:
>>     # This program is not thread safe so we must protect it from being
>>     # trampled over by another copy
>>     # pause if another email is being processed for half an hour maximum
>>     t = time.time()+1800
>>     locked = True
>>     while locked and time.time() < t:
>>         try:
>>             os.mkdir('instancelck')
>>             locked = False
>>             pass
>>         except :
>>             time.sleep(1)
>>             pass
>>         pass
>>     # do everything else
>>     .......
>> finally :
>>     os.rmdir('instancelck')
>>     # Removes the thread locking device so that another copy may run
>>     pass
>>
>> The timer was in case for any reason finally did not run successfully 
>> ever.
>>
>> Facundo's solution I have not yet studied.
>>
>> Thanks
>>
>> Nigel
>>
>>
> 
> Nigel,
> 
> So, basically you have a working solution.  Couple of things, though.  
> You may want to modify your
> 
> finally:
>    os.rmdir()
> 
> to
> 
> finally:
>    if not Locked:
>       os.rmdir()
> 
> otherwise, if one of them times out, it's going to delete that directory.
> Also, you probably want to tighten that except statement down to "except 
> OSError" so that you are handling  the only exception that should be 
> raised right in that block of code.  If something goes batty and raises 
> another type of exception, you probably want to fall into your filly 
> condition, log it (are you logging with this application?), then let 
> finally raise the exception up.
> 
> This last one is just a matter of preference, but you could remove most 
> of those "pass" statements.  They're not hurting anything, but it's just 
> a little cluttered.
> 
> OK - not I know that you weren't looking for comments on your code, but 
> you got it for free, right?  ;-)  Anyway, to your question.  Is there a 
> better way of doing this than the way you are doing it?  Probably, but I 
> don't know for sure what a good answer is.  Frankly, this approach both 
> scares the pants off of me and gives me an upset stomach.  I'm guessing 
> you must be feeling at least a little of that sentiment or you wouldn't 
> be asking if there's another way to do this.  I've just seen this type 
> of thing abused and lead to all kinds of weird and undiagnosable 
> problems.  When you're using the filesystem as either a locking 
> mechanism or as a source of state information, that just smells like the 
> wrong solution to me, but I'm probably biased against it from experience.
> 
> So, what are the alternatives.  This may be way over kill, but maybe you 
> could have procmail kick off a web-services-ish script and talk to a 
> centralized server process (either XMLRPC or SOAP or something like 
> that).  Then you could put the locking in the server process and not 
> have to deal with it from the procmail-spawned scripts.  And, actually, 
> if you used something like SimpleXMLRPCServer, you wouldn't have to do a 
> thing.  The first request would be handled and processed, while the 
> second and subsequent requests would block until the first (or next in 
> the case of subsequent requests) request finished.  But, how do you make 
> sure you've only got one XMLRPCServer running? ;-)
> 
> Another option is a recipe that is on the ASPN Python Cookbook site for 
> this type of thing:
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252495
> But at a glance it looked like more of the same of what you've already 
> done.
> 
> Another alternative is to use the builtin procmail locking as others 
> have suggested.  I googled on it and the most frequent recurring word 
> pattern was "procmail lock failure", so I'd be a little afraid of that.  
> But it'd be worth looking into nonetheless.
> 
> I don't know if in *NIX OSes if you can somehow register an instance of 
> a process and make any subsequent registration block execution until the 
> running one terminates.  That would be ideal.
> 
> Anyway, these are my thoughts.
> 
> 
> Jeremy Jones