Jeremy Hylton : weblog : 2003-12-19

Concurrent Updates in Versions Fixed

Friday, December 19, 2003

I fixed the two remaining tests on the MVCC branch today, which means we are very close to merging it with the trunk. There were actually two bugs, one trivial, the other somewhat tricky. The bugs were all in handling multiple revisions of objects in versions.

The easy bug was the MVCC code should not run in a connection that is using a version. We restricted the new loadBefore() method to only return non-version data in order to keep things simple. The connection was still trying to use it, but getting bogus non-version data. Easy to fix.

The trickier bug was provoked by the following scenario involving commits by two concurrent connections. Each connection is using a different version. They both operate on a BTree object locked in the first version. The 2nd-version connection loads and modifies the object, reading the non-version data.

Normally the 2nd-version transaction would fail with a locked version error when it commits, but in the failure case the first connection committed the version between the start and commit of the other transaction. That ordering produces a resolvable conflict, except that conflict resolution was seeing the version data for the original transaction when it should have seen the non-version data.

The conflict is resolvable because the commit version added a key and the current transaction is adding a different key. As a result, the merged tree should have both keys. If, on the other hand, the version data is seen for the original transaction, then it will appear like the committed transaction had no effect: it added a key that is already present. Conflict resolution will think that the conflicting transaction is okay and commit its state, erasing the key added by commit version.

It was a bug in loadSerial(). It returned version data when the serial number matched a versioned transaction. It should always return non-version data. One perplexing loose end is that it looks like the Berkeley storage has always had this bug. Perhaps Berkeley is so slow it never really provoked conflicts?

More hours burned supporting version trickery. Yesterday, Jim mentioned his latest idea for implementing versions without direct storage support. His idea is to use a wrapper or adapter that manages its own persistent storage to supplement the base storage. The wrapper implements the locking features necessary for versions -- or for an improved version-like feature. The benefit is that it only gets implemented once.

I think version locks need to be implemented at the storage level in order to get them right. I'm not sure if you can get the right effect at the application-level or not. (I think there was a thread about this on zodb-dev a year or so ago.)

The other big problem with tossing versions in the bin is that it would be a lot of work to remove them from the code. It's certainly not in scope for ZODB 3.3.