Pickle caching objects?

[Peter J. Holzer]

On 2019-12-16 08:56:26 +0100, dieter wrote:
> Note also that Python memeory management is quite elaborate:
> not every memory block is immediately obtained from and released
> to the operating system: Python has its own memory management
> data structures (to fill the gap between the fine grained
> memory requirements of a Python application and the mostly crude memory
> management services the operating system supports out of hand).
> This means that usually, a memory block freed by the Python application
> is not returned to the operating system but maintained by Python's
> memory management to be reused later. As a consequence, operating system
> tools for memory monitoring usually cannot tell the amount of memory
> really used by the application.

This is all true even if python does nothing special[1]: The C library
itself manages memory like this. So if python just calls malloc and
free you will see this behaviour. In particular the GNU C library (used
on most Linux systems) serves all "small" allocations from a common
pool (called the "heap") which can only grow and shrink at one end and
requests "large" allocations directly from the OS. Since the python
interpreter tends to use lots of small objects, most of the allocations
come from the heap which can't be returned to the OS unless there is a
large unused chunk right at the end.

        hp

[1] I haven't looked at the source code, but ltrace reveals a lot of
small mallocs, so it probably doesn't.

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20191216/a1f69d3e/attachment.sig>