getting MemoryError with dicts; suspect memory fragmentation

[Emin.shopper Martinian.shopper]

Dear Experts,

I am getting a MemoryError when creating a dict in a long running
process and suspect this is due to memory fragmentation. Any
suggestions would be welcome. Full details of the problem are below.

I have a long running processing which eventually dies to a
MemoryError exception. When it dies, it is using roughly 900 MB on a 4
GB Windows XP machine running Python 2.5.4. If I do "import pdb;
pdb.pm()" to debug, I see that it is dying inside a method when trying
to create a dict with about 2000 elements. If I instead do something
like

  p setattr(self,'q',list([list(xxx)+list(xxx)+list(xxx)+list(xxx) for
xxx in self.data]))

inside the debugger, I can make the memory increase to about 1.5 GB
WITHOUT getting a memory error.

If instead I do something like

  p setattr(self,'q',dict([(xxx,xxx) for xxx in range(1400)]))

inside the debugger, I get a MemoryError exception. If instead I do
something like

  p setattr(self,'q',dict([(xxx,xxx) for xxx in range(1300)]))

inside the debugger, I get no Exception.

I infer that python is trying to allocate a bunch of contiguous space
for the dict and due to fragmentation it can't find the contiguous
space and therefore it gives a memory error.

  1. Does this sound plausible or could something else be causing the problem?
  2. Does anyone have suggestions on how to fix this?

Some time Googling brings up the following 2004 thread started by Evan Jones:

  http://mail.python.org/pipermail/python-dev/2004-October/049480.html

but I'm unable to find a solution to the problem I'm having.

Any help would be much appreciated.

Thanks,
-E