[Mailman-Developers] about qrunner and locking

[J C Lawrence]

On Fri, 8 Dec 2000 10:36:25 -0800 
Chuq Von Rospach <chuqui@plaidworks.com> wrote:

>> Background for those who don't know: zodb is the Zope Object
>> Database, ZEO is Zope Enterprise Objects.

> My only worry about this is adding enough complexity and overhead
> that mailman loses it's attractiveness to the small site.

I argue similarly.  To echo you Chuq, a primary goal should be
ability to integrate.  That't given I'm increasingly coming to
question the use of Python pickles in the first place, let alone use
of custom database implementations.  I don't see that the value is
there for the increased complexity and isolation of the system.  We
already have enough problems given the fact that that the membership
base is kept in a pickle that I don't see much reason to go further
down that rat hole.  Yes, pickles are nice -- for private data that
will never be seen or accessed by an external system.

>> We'd have to handle collisions for multiple qrunner processes,
>> potentially on separate machines.  One way that doesn't involve
>> locking shenanigans is to divide the hash space up and assign a
>> segment to each out-qrunner process.

> here's another way that should work: each record has a locking
> field in it. When qrunner wants to execute that item, it reads the
> field. If the field is NULL, it writes its ID (hwatever it is,
> guaranteed unique) into that locking field. It then waits a beat,
> and reads it back. if it reads back its own ID, it knows it owns
> the record and can execute it. If it reads back someone else's ID,
> it lost the lock, but someone else owns the record so it can skip
> it and move on.

You missed a few race conditions in there.  As I wrote earlier the
only NFS operation which is guaranteed to be unique across
implementation is creat().  Given that, lock files based on the hash
of the message would seem to be the answer.  If you can create a
lock file with with a filename based on the hash of the message, you
have rights to deliver it.  If not, well, someone else obviously has
those rights.

The problem that remains is lockfile aging.

> you can simulate atomic locks with a little thought and
> cooperative processes, by everyone writing to the store and then
> seeing who won. A LOT easier from and administrative view than
> partitioning hashes and the like, IMHO.

This assumes that writes are atomic.  The problem is that they
occassionally aren't.

-- 
J C Lawrence                                       claw@kanga.nu
---------(*)                        : http://www.kanga.nu/~claw/
--=| A man is as sane as he is dangerous to his environment |=--