Object Persistence Using a File System

[Nick Vatamaniuc]

Chris,

Interesting concept. But why is there a need for a human readable
object persistence that is x10 slower than pickle? In other words
present a good use case with a rationale (i.e. your "criteria" that you
mentioned). The only one I can think of so far is debugging.

Also some objects are inherently not human readable (they are large, or
just binary/array data for example), or you could simply end up having
so many of them (10GB worth of disk space) that just due to the volume
they will become not very readable and you would need some kind of a
custom query mechanism (or find+grep) to search through your objects if
you wanted to read/edit values in them.

In your code comments I saw that another reason is resistance to
corruption. I think that a database that is backed up regularly is
probably a better solution. Besides, sometimes you want your failure to
be dramatic (go down with a bang!) so it gets your attention and you
can restore everything with a backup. Otherwise, with a tens of
thousands of files, some of them might end up being corrupted before
your next filesystem check gets to them, by then, the corruption could
spread (your program would load it, perform  computations, persist the
wrong results and so on),  and you wouldn't even notice it.

Hope these comments help,
Nick V.



Chris Spencer wrote:
> Before I get too carried away with something that's probably
> unnecessary, please allow me to throw around some ideas. I've been
> looking for a method of transparent, scalable, and human-readable object
> persistence, and I've tried the standard lib's Shelve, Zope's ZODB,
> Divmod's Axiom, and others. However, while they're all useful, none
> satisfies all my criteria. So I started writing some toy code of my own:
> http://paste.plone.org/5227
>
> All my code currently does is transparently keep track of object changes
> without requiring any special coding on part of the user, and a function
> to convert an object to a file system hierarchy of folders and files.
> Please, let me know what you think.
> 
> Thanks,
> Chris