"High water" Memory fragmentation still a thing?

[dieter]

Croepha <croepha at gmail.com> writes:
> Does python in general terms (apart from extensions or gc manipulation),
> exhibit a "high water" type leak of allocated memory in recent python
> versions (2.7+)?

Likely because it is very difficult to avoid. Would you need to
definitely prevent it is "memory compaction": not only is garbage
collected and freed but in addition, the used memory is also
relocated to form contigous blocks of used and free memory.

Without memory compaction, long running processes tend to suffer
from "memory fragmentation": while sufficient free memory is available,
it is available only in small blocks, not large enough for some memory
requests and those requests then call for more memory from the operating
system. When this memory is later released, it may become split up in
smaller blocks and when another large memory request arrives, new memory
may be requested from the operating system (even though the original
block would have been large enough).

Python tries hard to limit the effect of fragmentation (maintaining the
free blocks in bins of given size) but cannot eliminate it completely.


In order to support memory compaction, all C extensions must adhere
to a strict memory access protocol: they cannot freely use C pointers
to access the memory (as the compaction may invalidate those pointers
at any time) but must beforehand announce "I will now be using this pointer"
(such that the compaction does not move the memory) and afterwards
announce "I am no longer using this pointer" (such that memory
compaction becomes again possible for this memory).

As you see from the description, memory compaction presents a heavy burden
for all extension writers.