Locking around

[Ulrich Eckhardt]

Nikolaus Rath wrote:
> I need to synchronize the access to a couple of hundred-thousand
> files[1]. It seems to me that creating one lock object for each of the
> files is a waste of resources, but I cannot use a global lock for all
> of them either (since the locked operations go over the network, this
> would make the whole application essentially single-threaded even
> though most operations act on different files).

Just wondering, but at what time do you know what files are needed? If you
know that rather early, you could simply 'check out' the required files, do
whatever you want with them and then release them again. If one of the
requested files is marked as already in use, you simply wait (without
reserving the others) until someone releases files and then try again. You
could also wait for that precise file to be available, but that would
require that you already reserve the other files, which might unnecessarily
block other accesses.

Note that this idea requires that each access locks one set of files at the
beginning and releases them at the end, i.e. no attempts to lock files in
between, which would otherwise easily lead to deadlocks.

Further, you could distinguish between read-only and read-write access as an
optimisation.

> My idea is therefore to create and destroy per-file locks "on-demand"
> and to protect the creation and destruction by a global lock
> (self.global_lock). For that, I add a "usage counter"
> (wlock.user_count) to each lock, and destroy the lock when it reaches
> zero.
[...code...]

>  - Does that look like a proper solution, or does anyone have a better
>    one?

This should work, at least the idea is not flawed. However, I'd say there
are too many locks involved. Rather, you just need a simple flag and the
global lock. Further, you need a condition/event that tells waiting threads
that you released some of the files so that it should see again if the ones
it wants are available.

>  - Did I overlook any deadlock possibilities?

The normal deadlock possibilities when multiple locks are involved apply,
you must make sure that they are always acquired in an order that prevents
two threads waiting for a resource held by the other.

Uli

-- 
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932