writing large dictionaries to file using cPickle

[Aaron Brady]

On Jan 28, 10:13 am, perfr... at gmail.com wrote:
> hello all,
>
> i have a large dictionary which contains about 10 keys, each key has a
> value which is a list containing about 1 to 5 million (small)
> dictionaries. for example,
snip
> but this takes just as long... any ideas ? is there a different module
> i could use that's more suitable for large dictionaries ?
> thank you very much.

There is the 'shelve' module.  You could create a shelf that tells you
the filename of the 5 other ones.  A million keys should be no
problem, I guess.  (It's standard library.)  All your keys have to be
strings, though, and all your values have to be pickleable.  If that's
a problem, yes you will need ZODB or Django (I understand), or another
relational DB.