Why does shelve make such large files?

[Johan Wouters]

Gerrit Holl wrote:

> Ah, I understand.
> So pickle is useful for very small datases, but when they're really huge, one
> should use shelve. Isn't it?

Pickle will serialise your data into a certain format. This way you can
"store" objects like lists as a whole. The serialising has the benefit that
you can eg. put your object (be it a list or something else) through a pipe
or socket and depickle it at the other side. This way you can have python 
processes that exchange data in a sort of native python format.

Shelve gives you alot more than just storing the data as a long sequence.
Imagine the extra space being the administration that is needed for all
the extra functionality (like tagging what data items are valid etc)

The overhead induced by a database management system can be enormous
compared to the bare data you want to access, but this is mostly a
space/time tradeoff: the more space you use the faster you can do
things like searching and sorting etc etc.

kind regards,
===================================================================
Johan Wouters             ===              Easics               ===
ASIC Designer             ===  System-on-Chip design services   ===
Tel: +32-16-395 616          ===================================
Fax: +32-16-395 619      Interleuvenlaan 86, B-3001 Leuven, BELGIUM
mailto:johanw at easics.be           http://www.easics.com