[Tutor] using shelve

[Alan Gauld]

<chrispython at mac.com> wrote 

>I created a shelf called 'myshelf' and put two objects in it, 
> a string and a list. When I open the shelf I get:

>>>> d
> {'dir2': '/Users/development/Desktop/RSSReaderApp/RSS.db', 
> 'dir1': ['.DS_Store', '.localized', 'access the latest version', 
> 'Python Quick Reference.webloc', 'rssdb', 'workspace']}
> 
> It seems that when you use shelve.open(), it actually 
> brings the entire shelf dictionary into memory, 
> accessible through d.

No, it seems that when you evaluate d it does that.
Remember that when you type an object reference 
at the >>> prompt Python evaluates it and prints 
the repr() of that object. The repr() of a shelve requires 
it's contents be fetched.

> What if you had 100 objects in myshelf, or 1000 or 100,000? 
> Wouldn't it get bogged down?

That would depend on how much RAM you have and how 
big the objects were. If we say a typical modern PC has 
512M and the average shelve entry takes up 1KB (actually 
quite high) then you can store 100,000 objects in RAM 
with impuny.  But in ffact shelve doesn't store everything 
in RAM so if you don't ask for everything at once so you 
can go quite a bit higher than that.

> If so, what is it good for and how would you store a large 
> number of objects? (Storing strings and lists in an sql 
> database is probably the way to go for this simple 
> example, but what if  you had more complex objects?)

A SQL database is the usual solution for large collections 
of complex objects. Shelve is good for basic persistence 
but its not good for managing complex relationships, and 
in particular for retrieval of objects using complex search
criteria. Thats where SQL comes in. There are OODBs 
available too, of varying quality and they can run OQL 
which is the OO version of SQL and can base queries 
on methods as well as data. (Zope comes with an ODB 
but I have no idea how capable it is.) (The url just posted 
by Reed is excellent for reviewing the pros/cons of all the 
various Python storage mechanisms - thanks Reed)

However many, many applications really only need a basic 
form of persistence where objects are stored to disk 
between executions of the program and for that shelve 
(and even basic pickle) is perfectly adequate.

HTH,

Alan G.