Questions on Pickle and Shelve

[Peter Otten]

Virgil Stokes wrote:

> Here is snippet of Python (vers. 2.7.10) code that bothers me.
> 
> import cPickle as pickle
> 
> print "Pickle lists:"
> dogs = ['Rover','King','Spot','Rufus']
> cats = ['Mimi','Misty','Sasha']
> 
> with open('pickle.dat', 'wb') as pfile:
>      pickle.dump(dogs, pfile)
>      pickle.dump(cats,pfile)
> 
> del(dogs); del(cats)
> with open('pickle.dat', 'rb') as pfile:
>      dogs = pickle.load(pfile)
>      cats = pickle.load(pfile)
>      print dogs, '\n', cats, '\n'
> 
> import shelve
> 
> # Note! __exit__ attribute undefined for shelve
> sfile = shelve.open('shelve.dat')
> sfile['dogs'] = dogs
> sfile['cats'] = cats
> sfile.close()
> 
> print "Shelve entries:"
> del(cats); del(dogs)
> sfile = shelve.open('shelve.dat')
> #print sfile
> for key in sfile.keys():
>      print key,' - ',sfile[key]
> sfile.close()
> 
> 1)  Which (the pickle or shelve code) takes less total RAM, if dogs and
> cats were very large?

I think this is the wrong question. shelve is built on top of a key-value 
store. It can hold more cats and dogs than fit into memory, and lookup of a 
cat or dog should be efficient when you store one animal per key:

db = shelve.open(...)
db["Rover"] = Dog(...)
db["King"] =  Dog(...)
db["Misty"] = Cat(...)

If you put all dogs in a big list to store them in the shelve and there are 
only a few such lists there is no advantage over plain pickle.

> 2)  When the last shelve.open is given, is the entire contents of
> shelve.data
> transferred to RAM?  Note, if the print sfile is uncommented then the
> entire contents of shelve.data is printed out.

That is due to a careless implementation. Lookup is good, but everything 
else will only work for debugging, with small data samples.

> I was under the impression that the entire contents of a shelved file was
> not transferred to RAM when it was opened.

They aren't. The details vary depending on what database is used.