Persistent objects

[Alan Kennedy]

Hi Paul,

[Paul Rubin]
 > Basically I wish there was a way to have persistent in-memory objects
 > in a Python app, maybe a multi-process one.  So you could have a
 > persistent dictionary d, and if you say
 >    d[x] = Frob(foo=9, bar=23)
 > that creates a Frob instance and stores it in d[x].  Then if you
 > exit the app and restart it later, there'd be a way to bring d back
 > into the process and have that Frob instance be there.

Have you looked at Ian Bicking's SQLObject?

http://sqlobject.org/

To define a class

class MyPersistentObj(SQLObject):

	foo = IntCol()
	bar = IntCol()

To instantiate a new object

my_new_object = MyPersistentObj(foo=9, bar=23)

Once the new object has been created, it has already been persisted into 
a RDBMS table automatically. To reload it from the table/database, e.g. 
after a system restart, simply supply its id.

my_existing_object = MyPersistentObj.get(id=42)

Select a subset of your persistent objects using SQL-style queries

my_foo_9_objects = MyPersistentObj.select(MyPersistentObj.q.foo == 9)
for o in my_foo_nine_objects:
     process(o)

SQLObject also takes care of caching, in that objects are optionally 
cached, associated with a specific connection to the database. (this 
means that it is possible to have different versions of the same object 
cached with different connections, but that's easy to solve with good 
application architecture). So in your case, if your (web?) app is 
persistent/long-running, then you can simply have SQLObject cache all 
your objects, assuming you've got enough memory. (Hmm, I wonder if 
SQLObject could be made to work with weak-references?). Lastly, caching 
can be disabled.

I've found performance of SQLObject to be pretty good, but since you 
haven't specified particular requirements for performance, it's not 
possible to say if it meets your criteria. Although I feel comfortable 
in saying that SQLObject combined with an SQLite in-memory database 
should give pretty good performance, if you've got the memory to spare 
for the large databases you describe.

Other nice features include

1. RDBMS independent: currently supported are PostGres, FireBird, MySQL, 
SQLite, Oracle, Sybase, DBM. SQLServer support is in the pipepline. 
SQLObject code should be completely portable between such backend stores.

2. Full support for ACID transactional updates to data.

3. A nice facility for building SQL queries using python syntax.

4. Automated creation of tables and databases. Table structure 
modification supported on most databases.

5. Full support for one-to-one, one-to-many and many-to-many 
relationships between objects.

All in all, a great little package. I recommend that you take a close look.

Regards,

-- 
alan kennedy
------------------------------------------------------
email alan:              http://xhaus.com/contact/alan