Memory error while saving dictionary of size 65000X50 using pickle

Nagu thogiti at gmail.com
Mon Jul 7 15:05:47 EDT 2008


I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.

How do I go about resolving this? Is there way to partition the pickle
object and combine later if this is a problem due to limited resources
(memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).


Here is the detail description of the error:

Traceback (most recent call last):
  File "<pyshell#12>", line 1, in <module>
    s = pickle.dumps(itemsim)
  File "C:\Python25\lib\pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "C:\Python25\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python25\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python25\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python25\lib\pickle.py", line 663, in _batch_setitems
    save(v)
  File "C:\Python25\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python25\lib\pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "C:\Python25\lib\pickle.py", line 615, in _batch_appends
    save(x)
  File "C:\Python25\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python25\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "C:\Python25\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python25\lib\pickle.py", line 477, in save_float
    self.write(FLOAT + repr(obj) + '\n')
MemoryError: out of memory



More information about the Python-list mailing list