multiple instance on Unix

[Jeremy Jones]

Nigel King wrote:

> Jeremy,
> I have not explained very well what I wanted.
>
> I had a program that was called randomly by specific emails arriving 
> which asked for certain information. If two or more emails arrived 
> simultaneously then procmail asked two or more instances of my program 
> to run. These instances interfered with one another, so I needed a 
> process to stop that from happening. What my son devised was for the 
> first to create a directory and run and when finished to delete the 
> directory. The subsequent instances could try to create a directory 
> but fail in an atomic piece of code. They would sleep for 1 or more 
> seconds and then try again. The first of the subsequent instances that 
> tries and succeeds stops the others from succeeding.
>
> Now, this works but I wondered whether anybody knew of a more standard 
> bit of python code was available for ensuring that only one instance 
> was processing. mutex does it for threads but not for instances as I 
> understand it.
>
> The specification for a better process would include the ability to 
> ensure that the queue was orderly, in other words some sort of FIFO 
> would ensure that first served would have been the first to request 
> the lock and fail.
>
> Our solution which does not satisfy the previous paragraph.
>
> import os, time
> try:
>     # This program is not thread safe so we must protect it from being
>     # trampled over by another copy
>     # pause if another email is being processed for half an hour maximum
>     t = time.time()+1800
>     locked = True
>     while locked and time.time() < t:
>         try:
>             os.mkdir('instancelck')
>             locked = False
>             pass
>         except :
>             time.sleep(1)
>             pass
>         pass
>     # do everything else
>     .......
> finally :
>     os.rmdir('instancelck')
>     # Removes the thread locking device so that another copy may run
>     pass
>
> The timer was in case for any reason finally did not run successfully 
> ever.
>
> Facundo's solution I have not yet studied.
>
> Thanks
>
> Nigel
>
>

Nigel,

So, basically you have a working solution.  Couple of things, though.  
You may want to modify your

finally:
    os.rmdir()

to

finally:
    if not Locked:
       os.rmdir()

otherwise, if one of them times out, it's going to delete that directory. 

Also, you probably want to tighten that except statement down to "except 
OSError" so that you are handling  the only exception that should be 
raised right in that block of code.  If something goes batty and raises 
another type of exception, you probably want to fall into your filly 
condition, log it (are you logging with this application?), then let 
finally raise the exception up.

This last one is just a matter of preference, but you could remove most 
of those "pass" statements.  They're not hurting anything, but it's just 
a little cluttered.

OK - not I know that you weren't looking for comments on your code, but 
you got it for free, right?  ;-)  Anyway, to your question.  Is there a 
better way of doing this than the way you are doing it?  Probably, but I 
don't know for sure what a good answer is.  Frankly, this approach both 
scares the pants off of me and gives me an upset stomach.  I'm guessing 
you must be feeling at least a little of that sentiment or you wouldn't 
be asking if there's another way to do this.  I've just seen this type 
of thing abused and lead to all kinds of weird and undiagnosable 
problems.  When you're using the filesystem as either a locking 
mechanism or as a source of state information, that just smells like the 
wrong solution to me, but I'm probably biased against it from experience.

So, what are the alternatives.  This may be way over kill, but maybe you 
could have procmail kick off a web-services-ish script and talk to a 
centralized server process (either XMLRPC or SOAP or something like 
that).  Then you could put the locking in the server process and not 
have to deal with it from the procmail-spawned scripts.  And, actually, 
if you used something like SimpleXMLRPCServer, you wouldn't have to do a 
thing.  The first request would be handled and processed, while the 
second and subsequent requests would block until the first (or next in 
the case of subsequent requests) request finished.  But, how do you make 
sure you've only got one XMLRPCServer running? ;-)

Another option is a recipe that is on the ASPN Python Cookbook site for 
this type of thing:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252495
But at a glance it looked like more of the same of what you've already done.

Another alternative is to use the builtin procmail locking as others 
have suggested.  I googled on it and the most frequent recurring word 
pattern was "procmail lock failure", so I'd be a little afraid of that.  
But it'd be worth looking into nonetheless.

I don't know if in *NIX OSes if you can somehow register an instance of 
a process and make any subsequent registration block execution until the 
running one terminates.  That would be ideal.

Anyway, these are my thoughts.


Jeremy Jones