How to Buffer Serialized Objects to Disk

[Adam Tauno Williams]

On Wed, 2011-01-12 at 17:29 -0500, Scott McCarty wrote:
> Been digging ever since I posted this. I suspected that the response
> might be use a database.

I use shelve extensively; there are many use-cases where it makes sense.
And there are many where a database makes sense.

Basically, if I just want key based lookup and the data fits in memory I
use a shelve.

> The problem is I don't want any dependencies and I also don't need
> persistence program runs. I kind of wanted to keep the use of petit
> very similar to cat, head, awk, etc. But, that said, I have realized
> that if I provide the analysis features as an API, you very well,
> might want persistence between runs.
> What about using an array inside a shelve?
> Just got done messing with this in python shell:
> import shelve
> d = shelve.open(filename="/root/test.shelf", protocol=-1)
> d["log"] = ()
> d["log"].append("test1")
> d["log"].append("test2")
> d["log"].append("test3")
> Then, always interacting with d["log"], for example:
> for i in d["log"]:
>     print i
> Thoughts?

That is fine so long as all your data fits comfortable in memory.

> I know this won't manage memory, but it will keep the footprint down
> right?

No.  All of "log" will always be in memory.