Why custom objects take so much memory?

[Robert Kern]

Steven D'Aprano wrote:
> On Tue, 18 Dec 2007 21:13:14 +0100, Hrvoje Niksic wrote:
> 
>> Each object takes 36 bytes itself: 4 bytes refcount + 4 bytes type ptr +
>> 4 bytes dict ptr + 4 bytes weakptr + 12 bytes gc overhead.  That's not
>> counting malloc overhead, which should be low since objects aren't
>> malloced individually.  Each object requires a dict, which consumes
>> additional 52 bytes of memory (40 bytes for the dict struct plus 12 for
>> gc).  That's 88 bytes per object, not counting malloc overhead.
> 
> And let's not forget that if you're running on a 64-bit system, you can 
> double the size of every pointer.
> 
> Is there a canonical list of how much memory Python objects take up? Or a 
> canonical algorithm?

Here is Martin v. Löwis giving a few pointers (pardon the pun):

http://mail.python.org/pipermail/python-list/2002-March/135223.html
http://groups.google.de/group/comp.lang.python/msg/b9afcfc2e1de5b05?hl=de

> Or failing either of those, a good heuristic?

I thought there was a tool that tried to estimate this using hints from the
type, but Googling has availed me not.

>> Then there's string allocation: your average string is 6 chars long; add
>> to that one additional char for the terminating zero.
> 
> Are you sure about that?

Yes. Look at Include/stringobject.h:

"""
Type PyStringObject represents a character string.  An extra zero byte is
reserved at the end to ensure it is zero-terminated, but a size is
present so strings with null bytes in them can be represented.  This
is an immutable object type.
"""

> If Python strings are zero terminated, how does 
> Python deal with this?
> 
>>>> 'a\0string'[1]
> '\x00'

It stores a length separate from the value. The 0-termination is a courtesy to C
APIs that expect 0-terminated strings. It does not define the end of the Python
string, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco