Python garbage collection: not releasing memory to OS!

[Sam]

On 04/15/2016 05:25 AM, cshintov at gmail.com wrote:
> I have written an application with flask and uses celery for a long running task. While load testing I noticed that the celery tasks are not releasing memory even after completing the task. So I googled and found this group discussion..
>
> https://groups.google.com/forum/#!topic/celery-users/jVc3I3kPtlw
>
> In that discussion it says, thats how python works.
>
> Also the article at https://hbfs.wordpress.com/2013/01/08/python-memory-management-part-ii/ says
>
> "But from the OS's perspective, your program's size is the total (maximum) memory allocated to Python. Since Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase."
>
> And I use Linux. So I wrote the below script to verify it.
>
>      import gc
>      def memory_usage_psutil():
>          # return the memory usage in MB
>          import resource
>          print 'Memory usage: %s (MB)' % (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000.0)
>
>      def fileopen(fname):
>          memory_usage_psutil()# 10 MB
>          f = open(fname)
>          memory_usage_psutil()# 10 MB
>          content = f.read()
>          memory_usage_psutil()# 14 MB
>
>      def fun(fname):
>          memory_usage_psutil() # 10 MB
>          fileopen(fname)
>          gc.collect()
>          memory_usage_psutil() # 14 MB
>
>      import sys
>      from time import sleep
>      if __name__ == '__main__':
>          fun(sys.argv[1])
>          for _ in range(60):
>              gc.collect()
>              memory_usage_psutil()#14 MB ...
>              sleep(1)
>
> The input was a 4MB file. Even after returning from the 'fileopen' function the 4MB memory was not released. I checked htop output while the loop was running, the resident memory stays at 14MB. So unless the process is stopped the memory stays with it.
>
> So if the celery worker is not killed after its task is finished it is going to keep the memory for itself. I know I can use **max_tasks_per_child** config value to kill the process and spawn a new one. **Is there any other way to return the memory to OS from a python process?.**
>


With situations like this, I normally just fork and do the mem intensive 
work in the child and then kill it off when done. Might be  able to use 
a thread instead of a fork. But not sure how well all that would work 
with celery.

--Sam