"High water" Memory fragmentation still a thing?

[Steven D'Aprano]

Chris Angelico wrote:

> On Sat, Oct 4, 2014 at 11:02 AM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:
>> Chris Angelico wrote:
>>
>>> In theory, a high
>>> level language like Python would be allowed to move objects around to
>>> compact memory, but CPython doesn't do this, and there's no proof that
>>> it'd really help anything anyway.
>>
>> I welcome correction, but I understand that both the JVM and the CLR
>> memory managers move memory around. That's why Jython and IronPython use
>> sequential integers as object IDs, since memory locations are not fixed.
> 
> Right; I should have made it clearer that there's no proof that it'd
> help anything *in CPython*. Removing the GIL is periodically proposed,
> too, but there's no proof that its removal would benefit CPython; it's
> not just that nobody's gotten around to writing a memory compactor for
> CPython.

I think that you're conflating a couple of different issues, although I
welcome correction.

I don't think that removing the GIL is a requirement for a memory compactor,
or vice versa. I think that PyPy is capable of plugging in various
different garbage collectors, including some without the GIL, which may or
may not include memory compactors. So as far as I can tell, the two are
independent.

As far as the GIL in CPython goes, there have been at least two attempts to
remove it, and they do show strong improvements for multi-threaded code
running on multi-core machines. Alas, they also show significant *slowdown*
for single-core machines, and very little improvement on dual-core
machines.

[Aside: The thing that people fail to understand is that the GIL is not in
fact something which *prevents* multi-tasking, but it *enables* cooperative
multi-tasking:

http://www.dabeaz.com/python/GIL.pdf

although that's not to say that there aren't some horrible performance
characteristics of the GIL. David Beazley has identified issues with the
GIL which suggest room for improving the GIL and avoiding "GIL battles"
which are responsible for much of the overhead of CPU-bound threads. Any C
programmers who want to hack on the interpreter core?]


Nevertheless, you're right that since nobody has actually built a version of
CPython with memory compactor, there's no *proof* that it would help
anything.


-- 
Steven