[New-bugs-announce] [issue22293] mock could be less memory-intensive

[James Westby]

New submission from James Westby:

Hi,

I'm looking in to the memory usage of our testsuite, which does a fair amount of

  def setUp():
      patcher = patch.multiple(...)
      self.mock_whatever = patcher.start()
      self.addCleanup(patcher.stop)

or other ways of creating a mock and assigning it to an instance variable on a TestCase.

This means that by the end of a run, we have quite a lot of references to MagicMocks.

This then becomes the majority of the memory usage of the python process: (from meliae)

Total 1157176 objects, 1189 types, Total size = 327.1MiB (342972340 bytes)
 Index   Count   %      Size   % Cum     Max Kind
     0  575750  49 198058000  57  57     344 MagicProxy
     1   49955   4  52034888  15  72  196888 dict
     2  124127  10  11881628   3  76  386477 str
     3   12997   1  11749288   3  79     904 type
     4    8225   0   9146200   2  82    1112 MagicMock
     5   66310   5   5282392   1  84   80056 tuple
     6   38161   3   4579320   1  85     120 function
     7    1503   0   3972281   1  86   49488 module
     8   28506   2   3648768   1  87     128 code
     9   25768   2   2869680   0  88   69168 list
    10   12649   1   2803196   0  89   66356 unicode
    11    2251   0   2503112   0  89    1112 ClientHandler
    12    2228   0   2477536   0  90    1112 _patch
    13   28223   2   2257840   0  91      80 instancemethod
    14    2014   0   2239568   0  91    1112 BoundMethodWeakref
    15    2003   0   2227336   0  92    1112 DummyCache
    16   24681   2   2221112   0  93     792 _CallList
    17   18555   1   1632840   0  93      88 weakref
    18    1457   0   1550248   0  94    1064 Morsel
    19   46258   3   1110192   0  94      24 int

The fact that each MagicMock creates 72 MagicProxies means that it is a significant chunk of memory, despite each being small.

John Arbash Meinel suggested setting __slots__ = ['name', 'parent'] on MagicProxy to reduce the memory usage of this simple object.

This helps with the memory usage:

Total 1140377 objects, 1189 types, Total size = 169.5MiB (177755867 bytes)
 Index   Count   %      Size   % Cum     Max Kind
     0   47744   4  51347840  28  28  196888 dict
     1  574210  50  36749440  20  49      64 MagicProxy
     2  122020  10  11769659   6  56  386477 str
     3   12975   1  11729400   6  62     904 type
     4    8203   0   9121736   5  67    1112 MagicMock
     5   64125   5   5141368   2  70   80056 tuple
     6   36024   3   4322880   2  73     120 function
     7    1503   0   3972281   2  75   49488 module
     8   28506   2   3648768   2  77     128 code
     9   12643   1   2801540   1  79   66356 unicode
    10   23634   2   2716064   1  80   69168 list
    11    2251   0   2503112   1  82    1112 ClientHandler
    12   28223   2   2257840   1  83      80 instancemethod
    13    2014   0   2239568   1  84    1112 BoundMethodWeakref
    14    2003   0   2227336   1  85    1112 DummyCache
    15   24615   2   2214536   1  87     792 _CallList
    16   18482   1   1626416   0  87      88 weakref
    17    1457   0   1550248   0  88    1064 Morsel
    18   46259   4   1110216   0  89      24 int
    19    2496   0    858624   0  89     344 ModelState

I'm going to go through and drop references so that these can get garbage collected, but making Mock less memory-intensive would probably be appreciated by its users.

Reducing the memory usage of the tiny MagicProxies would be good, but also if there is a way to reduce the number of them by not pre-assiging 72 of them for every MagicMock, when each is very unlikely to be used, then that would be good as well.

Thanks,

James

----------
components: Library (Lib)
messages: 226017
nosy: james-w
priority: normal
severity: normal
status: open
title: mock could be less memory-intensive
type: resource usage
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22293>
_______________________________________