Jeremy Hylton : weblog : 2003-11-25

ZODB Serial Numbers

Tuesday, November 25, 2003, 1:07 a.m.

In the new multi-version concurrency control branch, the serial number seems to be growing irrelevant. Transaction ids are used directly in all the new code. Unfortunately, serial numbers are still necessary to support versions -- in particular, aborting a version.

The difference between a serial number and a transaction id only matters in a very small number of cases. Normally the serial number for each object in a transaction is the transaction's id. If the transaction is an abort version, then the objects get serial numbers that correspond to the serial number of the non-version data from an earlier transaction. One reason for this complication is to handle the original ZEO cache verification algorithm, which worked on serial numbers and didn't want to invalidate its current non-version data just because a version was aborted. The other reason is that a transaction that commits using the non-version data does not conflict with the abort version operation.

The only other place serial numbers are used is to detect conflicts. When a transaction stores a new revision of an object, it passes the previous serial number to the storage. If that previous serial number doesn't match the current serial number, a conflict has occurred -- another client has committed its changes first.

In several places for MVCC we need to know the starting and ending transaction ids for a revision. In most cases, this starting transaction is the same as the serial number. I wish there were some way to avoid the need for a separate serial number -- like a flag that was set only for abort version records. The server would have to do extra work to figure out whether a conflict really exist for an abort version, then. If we allowed abort version to generate spurious conflicts, we could simplify all the APIs.

In the details of the FileStorage implementation, serial numbers are tricky to manage because of backpointers. A data record contains a backpointer for undo and version operations, because they don't create a new copy of the data; they just point to an earlier record that does have the data. As a result, the implementation is really dealing with two data records, and you have to decide whether to use the serial number from the current record or the old record. FileStorage always uses the current serial number, which avoids a bunch of seeks+reads to resolve backpointers at store time. As a result, you have to copy the old serial number into the new record for abort version. Would it be possible to always pass the old serial number and then do a little more work for store calls?

The new loadNonCurrent call is returning the old serial number rather than the new one. That seems like the only sensible thing for Berkeley storages to do, but it's inconsistent with plain-old load.