pickling multiple dictionaries

[Hans Georg Krauthaeuser]

manstey wrote:
> Hi,
> 
> I am running a script that produces about 450,000 dictionaries. I tried
> putting them into a tuple and then pickling the tuple, but the tuple
> gets too big. Can I pickle dictionaries one after another into the same
> file and then read them out again?
> 
> Cheers,
> Matthew
> 
If you don't know, just try it:

In [1]:import pickle
In [2]:d1={'a':1}
In [3]:d2={'b':2}
In [4]:pfile=file('test.p','wb')
In [5]:pickle.dump(d1,pfile)
In [6]:pickle.dump(d2,pfile)
In [7]:pfile.close()
In [8]:del d1
In [9]:del d2
In [10]:pfile=file('test.p','rb')
In [11]:d1=pickle.load(pfile)
In [12]:d1
Out[12]:{'a': 1}
In [13]:d2=pickle.load(pfile)
In [14]:d2
Out[14]:{'b': 2}

If your data is *really* large, have a look to PyTables
(http://www.pytables.org/moin).

Regards,
Hans Georg