memory utilization blow up with dict structure

[Christian]

Hi,

I'm wondering why python blow up a dictionary structure so much.

The ids and cat substructure could have 0..n entries but in the most cases they are <= 10,t is limited by <= 6.

Thanks for any advice to save memory.
Christian


Example:

{'0a0f7a3a0e09826caef1bff707785662': {'ids': {'aa316b86-8169-11e6-bab9-0050563e2d7c',
 'aa3174f0-8169-11e6-bab9-0050563e2d7c',
 'aa319408-8169-11e6-bab9-0050563e2d7c',
 'aa3195e8-8169-11e6-bab9-0050563e2d7c',
 'aa319732-8169-11e6-bab9-0050563e2d7c',
 'aa319868-8169-11e6-bab9-0050563e2d7c',
 'aa31999e-8169-11e6-bab9-0050563e2d7c',
 'aa319b06-8169-11e6-bab9-0050563e2d7c'},
  't': {'type1', 'type2'},
  'dt': datetime.datetime(2016, 9, 11, 15, 15, 54, 343000),
  'nids': 8,
  'ntypes': 2,
  'cat': [('ABC', 'aa316b86-8169-11e6-bab9-0050563e2d7c', '74', ''),
   ('ABC','aa3174f0-8169-11e6-bab9-0050563e2d7c', '3', 'type1'),
   ('ABC','aa319408-8169-11e6-bab9-0050563e2d7c','3', 'type1'),
   ('ABC','aa3195e8-8169-11e6-bab9-0050563e2d7c', '3', 'type2'),
   ('ABC','aa319732-8169-11e6-bab9-0050563e2d7c', '3', 'type1'),
   ('ABC','aa319868-8169-11e6-bab9-0050563e2d7c', '3', 'type1'),
   ('ABC','aa31999e-8169-11e6-bab9-0050563e2d7c', '3', 'type1'),
   ('ABC','aa319b06-8169-11e6-bab9-0050563e2d7c', '3', 'type2')]},

   
I did a fresh read from pickled object to have a "clean" env.


linux-64bit:

sys.getsizeof(superdict)
50331744
len(superdict)
941272


VmPeak:	 2981364 kB
VmSize:	 2850288 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	 2108936 kB
VmRSS:	 1978076 kB
VmData:	 2541724 kB
VmStk:	     140 kB
VmExe:	       4 kB
VmLib:	  111136 kB
VmPTE:	    4380 kB
VmSwap:	       0 kB