Journal: a Prevayler-inspired persistence system

[Itamar Shtull-Trauring]

Yesterday just checked Journal, a persistence system inspired by
Prevayler into Twisted CVS (http://www.prevayler.org for details). The
idea is basically that all interactions with the system are done via
serializable commands objects, which are logged *before* they execute.
This allows us to keep down number of transactions in the system since
in the worst case, the system can be restored by running the logged
commands.

In the standard Prevayler model, the actual data model is stored just in
memory, with occasional snapshots serialized to disk (in Python this is
done by storing a pickle.) In case the snapshot is out of date or
corruped, you merely rerun the commands in the command log until your
data is once more up to date.

Additional benefits include being able to rerun sessions for testing
performances, changing the internal data model is easy (just delete
snapshot of system state and rerun all commands against new codebase),
and so on.

This is an alternative to the immutable database model of keeping old
versions of data - instead, we keep a log of all state changing commands
(or even queries - useful for cases when you need to keep track of every
read to the system).

The main difference from prevayler is flexibility - while I do support
the "everything in memory" model, I also have as yet untested code for
using RDBMS for storage as well, using twisted.enterprise.row (an
object/relational mapper). In the future Journal will support logging
commands to a RDBMS as well, allowing you to easily gather statistics
about everything that has ever been done to your system. While I haven't
tried the RDBMS code yet, I think the model is sound.

Plugging the code into any persistence mechanism for snapshotting
purposes should be possible, e.g. you could use ZODB as well. However
some will be more suited than others - t.e.row is pretty good since it
doesn't require a transaction for each command executed if integrated
correctly with Journal.

Example of using the system (with memory snapshotting and pickle based
log):
http://www.twistedmatrix.com/users/jh.twistd/viewcvs/cgi/viewcvs.cgi/twisted/test/test_journal.py?rev=1.3&content-type=text/vnd.viewcvs-markup&cvsroot=Twisted

ViewCVS of code:
http://www.twistedmatrix.com/users/jh.twistd/viewcvs/cgi/viewcvs.cgi/twisted/persisted/journal/?cvsroot=Twisted

Journal should be available in Twisted 1.0.1.

-- 
Itamar Shtull-Trauring    http://itamarst.org/
Available for Python, Twisted, Zope and Java consulting