shelve and nested dictionaries

[Fredrik Lundh]

Matthew Schibler wrote:

> I'm a newbie to Python, with some experience using perl (where I used
> nested arrays and hashes extensively). I am building a script in
> python for a MUD I play, and I want to use the shelve module to store
> persistent information between script executions. The following code
> does not work for me,
> 
> import shelve, sys, os, string
> db = shelve.open(os.path.abspath(os.path.dirname(sys.argv[0])) + '/' +
> 'sandbox.dat', 'c')
> db['JustSomeVariable'] = 'apple'
> db['subdb'] = {}
> db['subdb']['anotherdict'] = {}
> db['subdb']['anotherdict']['bleh'] = 'hello world'
> db.close()
> 
> of course, that's just a working example but it illustrates the
> problem i'm having. I think shelve objects act like dictionaries in a
> way, at least they seem to have dictionary keys beneath them.

the shelve module only tracks changes to the shelf itself (i.e. 
db[key]), not changes to to mutable objects stored in the shelve).

to change a mutable object, you have to fetch it, modify it, and then 
write it back:

     value = db[key]
     ... update value ...
     db[key] = value

in Python 2.3 and later, the shelve can help you with this, to some 
extent; from the help page:

   To avoid the problem with mutable entries, you may pass
   the keyword argument writeback=True in the call to
   shelve.open.  When you use:

           d = shelve.open(filename, writeback=True)

   then d keeps a cache of all entries you access, and writes
   them all back to the persistent mapping when you call
   d.close().  This ensures that such usage as
   d[key].append(anitem) works as intended.

   However, using keyword argument writeback=True may consume
   vast amount of memory for the cache, and it may make
   d.close() very slow, if you access many of d's entries
   after opening it in this way: d has no way to check which
   of the entries you access are mutable and/or which ones
   you actually mutate, so it must cache, and write back at
   close, all of the entries that you access.  You can call
   d.sync() to write back all the entries in the cache, and
   empty the cache (d.sync() also synchronizes the persistent
   dictionary on disk, if feasible).

</F>