Memory error while saving dictionary of size 65000X50 using pickle

[Larry Bates]

Nagu wrote:
> I am trying to save a dictionary of size 65000X50 to a local file and
> I get the memory error problem.
> 
> How do I go about resolving this? Is there way to partition the pickle
> object and combine later if this is a problem due to limited resources
> (memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).
> 
> 
> Here is the detail description of the error:
> 
> Traceback (most recent call last):
>   File "<pyshell#12>", line 1, in <module>
>     s = pickle.dumps(itemsim)
>   File "C:\Python25\lib\pickle.py", line 1366, in dumps
>     Pickler(file, protocol).dump(obj)
>   File "C:\Python25\lib\pickle.py", line 224, in dump
>     self.save(obj)
>   File "C:\Python25\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "C:\Python25\lib\pickle.py", line 649, in save_dict
>     self._batch_setitems(obj.iteritems())
>   File "C:\Python25\lib\pickle.py", line 663, in _batch_setitems
>     save(v)
>   File "C:\Python25\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "C:\Python25\lib\pickle.py", line 600, in save_list
>     self._batch_appends(iter(obj))
>   File "C:\Python25\lib\pickle.py", line 615, in _batch_appends
>     save(x)
>   File "C:\Python25\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "C:\Python25\lib\pickle.py", line 562, in save_tuple
>     save(element)
>   File "C:\Python25\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "C:\Python25\lib\pickle.py", line 477, in save_float
>     self.write(FLOAT + repr(obj) + '\n')
> MemoryError: out of memory

I've generated some rather large pickled dicts before and have never seen this 
before.  You can, of course, split your dictionary into several smaller ones, 
pickle them individually and combine them back together on unpickle/read using 
the dictionary update method.

BTW - 32-bit Windows can only address 3.5Gb of memory maximum.  If you have more 
installed, it is ignored.

-Larry