Ram memory not freed after executing python script on ubuntu system (rmlibre)

[Peter J. Holzer]

On 2020-05-29 14:28:59 +0900, Inada Naoki wrote:
> pymalloc manages only small blocks of memory.
> Large (more than 512 byte) memory blocks are managed by malloc/free.
> 
> glibc malloc doesn't return much freed memory to OS.

That depends on what "much" means.

Glibc does return blocks to the OS which it allocated via mmap, By
default, these are allocations larger than 128 kB.

So that means that

* Blocks smaller than 512 bytes are returned to the OS if the arena they
  are in is completely empty.

* Blocks between 512 bytes and 128 kB are not returned to the OS (unless
  they happen to be at the end of the heap).

* Blocks larger than 128 kB are returned to the OS:

Most Python objects are probably smaller than 512 bytes, so whether
"much" is returned to the OS mostly depends on whether arenas ever get
completely empty.

This stupid little test program returns memory to the OS quite nicely,
because it allocates and frees lots of objects together:

---8<------8<------8<------8<------8<------8<------8<------8<---
#!/usr/bin/python3
import time

a = []
for i in range(10):
    print('a', i)
    x = []
    for j in range(1000000):
        x.append(j)
    a.append(x)
    time.sleep(1)
    if i >= 5:
        print('d', i - 5)
        a[i - 5] = None
        time.sleep(1)
print('f')
---8<------8<------8<------8<------8<------8<------8<------8<---

(run it with strace to watch the mmap/munmap system calls)

A program with more random allocation patterns may suffer severe
internal fragmentation and end up with many mostly empty arenas.

> You can try jemalloc instead of glibc.

I think that would only make a difference if you have lots of objects
between 512 B and 128 kB. And only if those of similar size are
allocated and freed together.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20200529/88eb7440/attachment.sig>