low-end persistence strategies?

[John Lenton]

On Tue, Feb 15, 2005 at 06:57:47PM -0800, Paul Rubin wrote:
> I've started a few threads before on object persistence in medium to
> high end server apps.  This one is about low end apps, for example, a
> simple cgi on a personal web site that might get a dozen hits a day.
> The idea is you just want to keep a few pieces of data around that the
> cgi can update.
> 
> Immediately, typical strategies like using a MySQL database become too
> big a pain.  Any kind of compiled and installed 3rd party module (e.g.
> Metakit) is also too big a pain.  But there still has to be some kind
> of concurrency strategy, even if it's something like crude file
> locking, or else two people running the cgi simultaneously can wipe
> out the data store.  But you don't want crashing the app to leave a
> lock around if you can help it.
> 
> Anyway, something like dbm or shelve coupled with flock-style file
> locking and a version of dbmopen that automatically retries after 1
> second if the file is locked would do the job nicely, plus there could
> be a cleanup mechanism for detecting stale locks.
> 
> Is there a standard approach to something like that, or should I just
> code it the obvious way?

one easy way would be something along the lines of

    from ConfigParser import ConfigParser
    from fcntl import flock, LOCK_SH, LOCK_EX, LOCK_UN

    class LockedParser(ConfigParser):
        def _read(self, fp, fpname):
            flock(fp, LOCK_SH) # block until can read
            try:
                rv = super(LockedParser, self)._read(fp, fpname)
            finally:
                flock(fp, LOCK_UN)
            return rv

        def write(self, fp):
            flock(fp, LOCK_EX) # block until can write
            try:
                rv = super(LockedParser, self).write(fp)
            finally:
                flock(fp, LOCK_UN)
            return rv

although you could do the same kind of stuff with csv, or even
Pickle. Of course this doesn't work if what you're wanting to
implement is a hit counter, but that is much easier: just grab a
LOCK_EX, read in, write out, LOCK_UN. If you care about (not)
overwriting changes, but fear you'll hold the lock for too long with
the simple 'grab the lock and run' approach, you could save a version
of the original file and compare before writing out. Complexity grows
a lot, and you suddenly would be better off using pybsddb or somesuch.

Of course I'm probably overlooking something, because it really can't
be this easy, can it?

-- 
John Lenton (john at grulic.org.ar) -- Random fortune:
BOFH excuse #44:

bank holiday - system operating credits  not recharged
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20050217/e5a23de2/attachment.sig>