LWDB: Lightweight Database Objects for Python - Version 0.05

[Warren Postma]

This is my first significant Python effort. I am hoping it might be useful
to some other folks,
and so I'm releasing a preview (incomplete) so that anyone else who needs an
extremely lightweight
(embedded) database can tell me if they see some promise in it, or problems
with it, or both.

This all start when I decided Shelve and dbShelve were not the right way to
solve my problem.

I decided I really wanted a class object to represent Rows, and a variety of
Table objects that all
stored data differently, but which all had the same interface.

I did some poking around under the hood and settled on a dictionary-based
approach, but used
"marshal" instead of "pickle", and used the BSDDB 2.0 module instead of the
BSDDB 1.0 that came
with Python.

So what I have built is a wrapper on top of the BSDDB 2.x (from Sleepycat
Software) using BSDDB 2.0
wrappers by Robin Dunn (generated with SWIG).

I also settled on the excellent mxDateTime object to provide date-time
column types.

Before storing a row to a file or an in-memory Python dictionary (depending
on whether it is a disk or memory table), LWDB splits rows into two
marshalled strings, one of which is the dictionary key, and the other half
of which is rest of the columns in the row, which are already part of the
key.

It uses tonnes of extremely dynamic Python, especially the code to
dynamically create new column types (for lookup operations).  This code
could not have been written in a statically typed language, without
reinventing Python, pretty much.  I don't think I could have written these
objects in any other language, except perhaps LISP.

It's labeled version  0.05 because it's not yet feature complete (see
below), but I'd like to know if anyone else has a need for an
ultra-lightweight or embeddable Database Row and Table objects that are
small and fairly efficient, have low disk and memory overhead, and are
written entirely in Python + C Extensions.

I'm interested in hearing whether or not anyone else feels like none of the
existing database toolkits for Python are exactly what they need. I am using
this stuff in an embedded data-acquisition system which is about halfway to
completion.  All the other Python solutions were too large for me.

A test script is included to give an idea of the flavour of programming with
it, and to give some performance figures.  The script was written on Windows
but should run on any platform which you can run Python + mxDateTime +
BSDDB.  If you can't get it to run on your Windows system, send me an Email
and I'll gladly try to help.

I'm working on making it even smaller, faster, and lighter.  One way to
conserve disk space would be to use Zlib to compress the data rows further.
I'm also interested in dropping the BSDDB and implementing a btree or hash
storage directly to disk, using a C extension of my own crafting.  At that
point, I would say we could drop BSDDB 1.0 code right out of future releases
of Python, because it is buggy and unstable. BSDDB 2.0 has very restrictive
licensing.  Anyone else think a BSDDB workalike should be included (with a
liberal license) as source code, with Python?

I'm also working on some more useful sample applications, including a
command line utility script(repair, import and export) and a GUI utility in
wxPython to let you view, repair, import, export, and do data entry into
these tables.

I'd also like to implement a simple 'distributed database' layer on top of
these tables, that uses a "publish subscribe" model to have several
computers, loosely connected by dialup or ethernet, send updates to each
other as "events". I'm waiting for the Python SOAP support to mature before
I proceed to that level.

Download:
http://www.geocities.com/charanga.geo/lwdb-0_05-alpha.zip

Readme file (from the zip file):
http://www.geocities.com/charanga.geo/readme.txt

Warren Postma
ZTR Control Systems