Help: using msvcrt for file locking

[Bengt Richter]

On Sun, 26 Aug 2001 16:19:36 GMT, Sheila King <sheila at spamcop.net> wrote:
[...]
>For CGI, I need file locking. The alternatives are (as I understand
>them):
>
>1. Have a server running, which controls access to the files, thus
>ensuring that only one process can write to the files at a time.
>
>2. Something dealing with sockets, where the CGI script tries to see if
>a certain socket is listening to determine whether it has permission to
>write to the file...details are hazy and I'm not going to look this one
>up again right now.
>
>3. Check whether a certain file exists to determine whether or not the
>process can have permission to write to the files.
>
>The problems with # 1 and 2, is that it requires a process running on
>the web server all the time, and I am not allowed to do that on the
>shared, community server where I host my website. (I don't think most
>people who do not run a dedicated server are allowed to have processes
>running in the background on the server.)
>
>The problem with #3, is I've heard that is unreliable as well. Problems
>with whether the file will be deleted or written if the program crashes.
>Then, if the file is still there/missing, subsequent calls to the CGI
>script will think that they have permission to write when they don't.
>Or, the process may think that it can't have permission, when it can.
>Etc...
>
>Since in a CGI situation, you may have many processes running in
>parallel, then "my program" doesn't control access to the files, since
>there may be many instances of the program. Actually, this is what I
>thought to do originally, but After going over # 1, 2, 3 above, I
>thought...gee, this isn't going to work. Then, I found a really nifty
>example in Mark Lutz Programming Python, Chapter 14 where he does a
>shelf database implementation for his Book Errata website. He discusses
>file locking in there, but only for a Unix-type platform.
>
>I really don't see any way around file locking for the type of
>application I'm trying to write. However, I guess after seeing all the
>trouble with the msvcrt module, and how it doesn't even seem to allow
>shared locking access to a file, that I will go ahead and use the
>win32all extensions. I'm supposing that will allow better control?
>

I'll assume that when you say 'file locking', you mean the file as
a whole, rather than a region of the file, as the parameters for _locking
specify (I think an example of yours had parameters for the whole
file length).

Have you considered using os.rename() to implement your method #3 above?
Barring a system crash, renaming should be atomic, and even if there
is a crash after renaming, you should be able to detect orphan files
if you put appropriate info in the new name.

I'll stick my neck out here with a pretty untested sketch:

Here is a sample interactive sequence (there's a file 'xxx'
in the current directory) ...

 Python 2.1 (#15, Apr 16 2001, 18:25:49) [MSC 32 bit (Intel)] on win32
 Type "copyright", "credits" or "license" for more information.
 >>> from LockedFile import LockedFile
 >>> flock=LockedFile('xxx','r')
 >>> if flock.open():
 ...     flock.f.read()
 ... else:
 ...     print flock.lasterror
 ...
 'This is xxx with no extension\n'
 >>> glock=LockedFile('xxx','r')
 >>> if glock.open():
 ...     glock.f.read()
 ... else:
 ...     print glock.lasterror
 ...
 [Errno 13] Permission denied trying to lock "xxx.998881229"
 >>> del flock
 >>> if glock.open():
 ...     glock.f.read()
 ... else:
 ...     print glock.lasterror
 ...
 'This is xxx with no extension\n'
 >>> glock.lockedName
 'xxx.998881516'
 >>> glock.f
 <open file 'xxx.998881516', mode 'r' at 007F74F0>
 >>> del glock

That Errno 13 was because I was slow, and the locked
file looked orphaned after 10 seconds, but it was still
open via the other open.

Note that calling glock.open() doesn't do anything if
it's already open, except clear the last error. If it
was locked but not orphaned yet, it should return None,
but with no lasterror (needs testing). Thus you might
be able to do glock.open() and sleep a little to let others free
the file. 10 seconds is a long time though, but be careful
with 1-second resolution, since a difference of 1 can mean
a microsecond or a microsecond short of two seconds, depending
on when the ticks happened in reality.

There's a mutex module that might be of interest too.
And you might want to change this thing to use all lowlevel
io instead of the builtin open and file object. You might
be able to get shared reading without error.

I don't know how/whether this incomplete workaround thing
fits your use, but HTH get you closer.

________________________________________________________

# LockedFile.py -- a class for locking by renaming, then opening a file
# Copyright (C) 2001 Bengt Richter. All rights reserved.
# Released under Python license, with NO WARRANTIES EXPRESSED OR IMPLIED.
# Version 0.01 alpha -- just a sketch, rework to suit ;-)
#
MAXLOCKSECONDS = 10 # max time to hold lock, in seconds (resolution 1 sec)

import os, glob, time
class LockedFile:
    def __init__(self, fullPath, fmode='r',maxhold=MAXLOCKSECONDS):
        self.lasterror = None
        self.f = None
        self.fdir = fullPath.split(os.sep)
        self.fname = self.fdir[-1]
        self.fdir = os.sep.join(self.fdir[:-1])
        if self.fdir:
            self.fdir += os.sep
        self.fmode = fmode
        self.open()

    def open(self):
        self.lasterror = None
        if self.f: return self.f

        self.lockedName = self.fname+'.'+str(long(time.time()))
        fpath = self.fdir+self.fname    #pathname for unlocked file
        
        if not os.access(fpath,os.F_OK):    # just checks existence

            # check for orphan over MAXLOCKSECONDS seconds old
            oldDir = os.getcwd()
            if self.fdir: os.chdir(self.fdir) # want names w/o paths
            orphan = glob.glob(self.fname+'.*')
            os.chdir(oldDir)

            if len(orphan) != 1:
                self.lasterror = '%d items matched "%s.*' % (len(orphan),self.fname)
                return None
            orphan = orphan[0]
            torph = orphan.split('.')[-1]
            if not torph.isdigit():
                self.lasterror = '"%s" has bad lock suffix' % orphan
                return None
            torph = long(torph)
            tnow = long(time.time())
            if tnow-torph > MAXLOCKSECONDS:
                # assume ok to take over orphan locked file
                fpath = self.fdir + orphan
            else:
                return  None #no, error, just locked, so f remains None
        
        # now try to lock the file at fpath
        try:
            os.rename(fpath, self.lockedName)
            self.f = open(self.fdir+self.lockedName, self.fmode)
            return self.f
        except OSError,e:
            self.lasterror = str(e) +' trying to lock "%s"' % fpath
        
    def close(self):
        if self.f:
            self.f.close()   # close the locked file
            self.f = None
            # try to rename it to its unlocked name
            try:
                os.rename(self.fdir+self.lockedName, self.fname)
            except OSError,e:
                self.lasterror = e.errmsg
            

    def __del__(self):
        self.close()
________________________________________________________