memory usage

[Nagy Gabor]

On 03-May-06 16:33, Troels Therkelsen wrote:
> In article <mailman.1052227330.15773.python-list at python.org>, Nagy Gabor wrote:
> > class TD:
> >     def __init__(self):
> >         self.Tag = T()
> >         self.Data = None
> > 
> 
> Since you write elsewhere in your post that these classes gets instantiated
> nearly 600000 times (I couldn't figure out if they stay in memory for the
> duration of the program but it would appear so),

Yes, thes stay.

> maybe you want to look into using __slots__ as this will help a lot on
> memory usage if you're having 2*600000 classes in memory. Eg.,
> 
> class TD:
>     __slots__ = ['Tag', 'Data']
>     def __init__(self):
>         self.Tag = T()
>         self.Data = None

I'll check this out.

> Alternatively, if you don't plan on further complicating T and TD, you might
> want to consider using a datatype with less overhead, like a list or a tuple.

Why do classes have more overhead than a list for example?
In my C++ background I knew exactly how much memory a class will take, and
it was not significantly bigger than the memory of the data members. Some
bytes for administration is OK.
However my experiments' results puzzle me:
If I don't write the actual data from the 2.3M file into the objects, the
memory usage drops from 250M to 120M. The data from the file were not
duplicated, so it seems that storing a bit less than 2.3M in 600k objects
takes 130M. :-(
If I don't append the tmp objects to the fields list, (so that the 600k
empty objects (in init: self.Tag=None, self.Data=None) are nowhere) the
memory usage dropt to 10M.

What I don't understand is this:
>>> fields = []
>>> class A:
...     pass
>>> for i in range( 1, 600000):
...     fields.append( A() )

When ran, the python2.2 process takes 116M of virtual memory. When
started, it took about 2M. When I appended i, the process took 11M.
So it seems when the element is a number, it takes roughly 20bytes, if it
is an object, it takes roughly 200bytes.

__slots__ seems to be a great step, the above example with new class and
with slots took only 29M.

So my biggest problem here is that I don't know what will python keep in
memory, (the overhead) so I can't guess sizes in advance.

I will go on experimenting

Regards,
Gee