Persistence design [was: RE: OT: why are LAMP sites slow?]

[Jack Diederich]

On Fri, Feb 04, 2005 at 10:31:19AM -0800, Robert Brewer wrote:
> Jack Diederich wrote:
> > *ding*ding*ding*  The biggest mistake I've made most 
> > frequently is using
> > a database in applications.  YAGNI.  Using a database at all has it's
> > own overhead.  Using a database badly is deadly.  Most sites would
> > benefit from ripping out the database and doing something simpler.
> > Refactoring a database on a live system is a giant pain in the ass,
> > simpler file-based approaches make incremental updates easier.
> > 
> > The Wikipedia example has been thrown around, I haven't looked at the
> > code either;  except for search why would they need a database to
> > look up an individual WikiWord?  Going to the database 
> > requires reading
> > an index when pickle.load(open('words/W/WikiWord')) would 
> > seem sufficient.
> > 
> ...
> > 
> > If there is interest I'll follow up with some details on my own LAMP
> > software which does live reports on gigs of data and - you 
> > guessed it -
> > I regret it is database backed.  That story also involves why 
> > I started
> > using Python (the prototype was in PHP).
> 
> I'd be interested, if only selfishly to hear more potential use cases
> for *my* projects. ;)

It would be long so I'll spin it into a blog piece.  They seem to be 
all the rage these days *0.5 wink*
 
> One of my goals for Dejavu* (my ORM) is to abstract persistence to the
> point that you can easily test your actual, live dataset against many
> potential storage mechanisms (i.e. multiple DB's, but also shelve,
> etc.). I need to finish the migration tools, but it's well on the way.

I just looked at the docs, looks like a nicer version of the homespun
one I use.  If someone lends you the time machine set the year to 2001
and I'll use Dejavu instead. Reading the docs also makes me long for
class decorators (again).  I use metaclasses to register tables to
databases, I see you inspect globals.

ex in a better world:

mydb = Database('foo')

@mydb.add_table
class Foo(Table):
  col1 = int
  col2 = str

@mydb.add_table
class Bar(Table):
  foo_id = str
  colb = unicode

*sigh* maybe in 2.5?

-Jack