[Tutor] object size in python is in what units?

[eryksun]

On Tue, Jul 23, 2013 at 12:19 PM, Oscar Benjamin
<oscar.j.benjamin at gmail.com> wrote:
> On 23 July 2013 16:25, eryksun <eryksun at gmail.com> wrote:
>>
>> 3.3 also switched to using anonymous mmap on POSIX (where supported),
>> and as your link shows -- 3.4 looks to be switching to VirtualAlloc on
>> Windows.
>
> Why is it better to use mmap than malloc?
>
> I read a few things about this but the main thing I found seemed to be
> that on systems supporting mmap, malloc itself will use mmap for
> sufficiently large enough allocations. That would seem to indicate
> that CPython would be better off letting the system decide when to use
> it (I'm not arguing with the code, just confused).

glibc malloc, for example, starts with a threshold at 128 KiB. But it
adjusts this upward when you free larger blocks, up to
DEFAULT_MMAP_THRESHOLD_MAX, which is 512 KiB on 32-bit systems and 4
MiB on 64-bit systems (as "man mallopt" informs me).

That means in practice Python's 256 KiB small object arenas end up on
the heap. That's subject to a dangling object allocated in the topmost
arena (object references are pointers, so it can't be automatically
moved to defragment the heap), even if (a big if) glibc would
otherwise trim the heap. But also Python is already managing the
arenas. Isn't it wasteful to have the C runtime also incurring
overhead for these blocks? Reducing peak memory usage by a small
amount (2-3%) appears to be the argument for the 3.4 switch to
VirtualAlloc on Windows.

mmap patch
http://bugs.python.org/issue11849

VirtualAlloc patch
http://bugs.python.org/issue13483