Memory usage per top 10x usage per heapy

[Oscar Benjamin]

On 25 September 2012 19:08, Junkshops <junkshops at gmail.com> wrote:

>
>  Can you give an example of how these data structures look after reading
> only the first 5 lines?
>
> Sure, here you go:
>
> In [38]: mpef._ustore._store
> Out[38]: defaultdict(<type 'dict'>, {'Measurement':
> {'8991c2dc67a49b909918477ee4efd767':
> <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0fe90>,
> '7b38b429230f00fe4731e60419e92346':
> <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0fad0>,
> 'b53531471b261c44d52f651add647544':
> <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0f4d0>,
> '44ea6d949f7c8c8ac3bb4c0bf4943f82':
> <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0f910>,
> '0de96f928dc471b297f8a305e71ae3e1':
> <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0f550>}})
>

Have these exceptions been raised from somewhere before being stored? I
wonder if you're inadvertently keeping execution frames alive. There are
some problems in CPython with this that are related to storing exceptions.


>
> In [39]:
> mpef._ustore._store['Measurement']['b53531471b261c44d52f651add647544'].typeStr
> Out[39]: 'Measurement'
>

Seeing how long these hex strings are, I'm confident that you would save a
significant amount of memory by converting them to int.


>
> In [40]:
> mpef._ustore._store['Measurement']['b53531471b261c44d52f651add647544'].lineNumber
> Out[40]: 5
>
> In [41]: mpef._ustore._idstore
> Out[41]: defaultdict(<class 'micropheno.exchangeformat.KBaseID.IDStore'>,
> {'Measurement': <micropheno.exchangeformat.KBaseID.IDStore object at
> 0x2f0f950>})
>
> In [43]: mpef._ustore._idstore['Measurement']._SIDstore
> Out[43]: defaultdict(<function <lambda> at 0x2ece7d0>, {'emailRemoved':
> defaultdict(<function <lambda> at 0x2c4caa0>, {'microPhenoShew2011':
> defaultdict(<type 'dict'>, {0: {'MLR_124572462':
> '8991c2dc67a49b909918477ee4efd767', 'MLR_124572161':
> '7b38b429230f00fe4731e60419e92346', 'SMMLR_12551352':
> 'b53531471b261c44d52f651add647544', 'SMMLR_12551051':
> '0de96f928dc471b297f8a305e71ae3e1', 'SMMLR_12550750':
> '44ea6d949f7c8c8ac3bb4c0bf4943f82'}})})})
>

Also I think lambda functions might be able to keep the frame alive. Are
they by any chance being created in a function that is called in a loop?

>>> def f():
...     x = 4
...     return lambda : x
...
>>> g = f()
>>> g()  # Accesses local variable from kept-alive frame
4
>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

Oscar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20120925/0d410435/attachment.html>