hard memory limits

[John Machin]

On Fri, 06 May 2005 18:24:21 +1000, Maurice LING <mauriceling at acm.org>
wrote:

>Hi,
>
>I think I've hit a system limit in python when I try to construct a list 
>of 200,000 elements. My error is
>
>malloc: vm_allocate (size = 2400256) failed......
>
>Just wondering is this specific to my system or what? Will adding more 
>RAM helps in this case?

Not if it's an OS limit (see other posts). Not if you are doing
something so weird or drastic that you will use up the extra RAM and
still get the same message.

If you were to reply to Fredrik's question (HOW are you creating your
list), and this one: WHAT is an "element", we might be able to help
you avoid a trip to the Apple dealer.

As a bit of a reality check for you:
[numbers based on a 32-bit machine, CPython]

An extra list of 200000 ints will take up 800000 bytes (plus small
change) if the ints are all in range(-1, 101) and thus cached --
that's a 4-byte pointer (i.e. PyObject *) each.

If all the ints are outside that range, and are distinct, they'll take
(worst case) 16 bytes each (assuming you aren't using a debug build of
Python). The additional 12 bytes are for the int object: a reference
counter, a pointer to the type object, and the actual value. So you're
looking at approx 3.2MB.

In general, reckon on each element taking up 12+sizeof(element_value).

I'd suspect that your "elements" are not quite as elementary as ints,
and/or you are doing a whole lot of *other* memory allocation. Confess
all ...

Cheers,
John