Pickle caching objects?

[Chris Angelico]

On Sun, Dec 1, 2019 at 12:15 PM José María Mateos <chema at rinzewind.org> wrote:
>         print("Memory usage:", psutil.Process(os.getpid()).memory_info().rss)
>
> Notice that memory usage increases noticeably specially on files 4 and
> 5, the biggest ones, and doesn't come down as I would expect it to. But
> the loading time is constant, so I think I can disregard any pickle
> caching mechanisms.
>
> So I guess now my question is: can anyone give me any pointers as to why
> is this happening? Any help is appreciated.
>

I can't answer your question authoritatively, but I can suggest a
place to look. Python's memory allocator doesn't always return memory
to the system when the objects are freed up, for various reasons
including the way that memory pages get allocated from. But it
internally knows which parts are in use and which parts aren't. You're
seeing the RSS go down slightly at some points, which would be the
times when entire pages can be released; but other than that, what
you'll end up with is a sort of high-water-mark with lots of unused
space inside it.

So what you're seeing isn't actual objects being cached, but just
memory ready to be populated with future objects.

ChrisA