Ram memory not freed after executing python script on ubuntu system (rmlibre)

[rmlibre at riseup.net]

We just ran into this problem when running our aiootp package's memory
hard password hashing function (https://github.com/rmlibre/aiootp/). The
memory was not being cleared after the function finished running but the
script was still live. We tried making sure everything went out of scope
and deleting things explicitly inside the function, but that didn't
help. We even tried forcing the garbage collector to free up
unreferenced memory with import gc; gc.collect(). But that only had a
small dent in the memory that was being built up.

The most useful answer online had to do with Python's free lists being
created automatically when very large datasets were being processed
(https://stackoverflow.com/questions/23937189/how-do-i-use-subprocesses-to-force-python-to-release-memory/24126616#24126616).


After putting the memory intensive work into a separate process, as the
answer suggested, the memory was finally freed after execution.

In our case, we wound up passing the result back in a
``multiprocessing.Manager().List()``. Though, passing your whole numpy
dataset back doesn't seem feasible. I'd recommend doing the necessary
memory intensive work in a separate process and passing only the
necessary conclusions back to the main process. 

Or you could set up a multiprocessing queue to pass control messages to
the spawned (daemon) process so it returns desired results on demand. 


Here's an excerpt from our code after the fix in case it's helpful.

>    @classmethod
>    def passcrypt(cls, password, salt, kb=1024, cpu=1024, hardness=256):
>        """
>        The ``passcrypt`` function can be highly memory intensive.
>        These resources may not be freed up, & often are not, because of
>        python quirks around memory management. This is a huge problem.
>        So to force the release of those resources, we run the function
>        in another process which is guaranteed to release them.
>        """
>        cls._validate_passcrypt_args(kb, cpu, hardness)
>        state = Manager().list()
>        process = Process(
>            target=cls._passcrypt,
>            args=(password, salt),
>            kwargs=dict(kb=kb, cpu=cpu, hardness=hardness, state=state),
>        )
>        process.start()
>        process.join()
>        return state.pop()




On 2020-05-28 16:00, python-list-request at python.org wrote:
> Send Python-list mailing list submissions to
> 	python-list at python.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://mail.python.org/mailman/listinfo/python-list
> or, via email, send a message with subject or body 'help' to
> 	python-list-request at python.org
> 
> You can reach the person managing the list at
> 	python-list-owner at python.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-list digest..."
> 
> Today's Topics:
> 
>    1. Re: Custom logging function (zljubisic at gmail.com)
>    2. Ram memory not freed after executing python script on ubuntu
>       system (Rahul Gupta)
>    3. Re: Ram memory not freed after executing python script on
>       ubuntu system (Chris Angelico)
>    4. Re: Custom logging function (Peter Otten)
>    5. Re: Elegant hack or gross hack? TextWrapper and escape codes
>       (Peter Otten)
>    6. Re: Elegant hack or gross hack? TextWrapper and escape codes
>       (Chris Angelico)
>    7. Re: Elegant hack or gross hack? TextWrapper and escape codes
>       (Peter Otten)
>    8. Re: Ram memory not freed after executing python script on
>       ubuntu system (Rahul Gupta)
>    9. Re: Ram memory not freed after executing python script on
>       ubuntu system (Chris Angelico)
>   10. Re: Behaviour of os.path.join (BlindAnagram)
>   11. Constructing mime image attachment (Joseph L. Casale)
>   12. Re: Behaviour of os.path.join (Eryk Sun)
>   13. Re: Behaviour of os.path.join (Eryk Sun)
>   14. Re: Behaviour of os.path.join (BlindAnagram)
>   15. Re: Behaviour of os.path.join (Eryk Sun)