memory consumption

Alexey zen.supagood at gmail.com
Thu Apr 1 07:50:39 EDT 2021


Found it. As I said before the problem was lurking in the cache.
Few days ago I read about circular references and things like that and
I thought to myself that it might be the case. To build the cache I was
using lots of 'setdefault' methods chained together

self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})...

and instead of wring a long lines I decided to divide it to increase 
readability

cluster = self.__cache.setdefault(cluster_name, {})
database = database.setdefault(database_name, {})
...
and I guess that was the problem.

First thing I did was to rewrite this back to single line. And it helped.
In the morning I tried different approach and decided to clear cache
with different way. So instead of doing self.__cache.clear(), 
self.__cache = None or even 'del self.__cache' I did:

for item in list(self.__cache.keys()):
	del self.__cache[item]

and againg effect was positive. As a result I decided to rewrite all the
methods to build,update and get from cache without 'setdefault' and 
to use "for loop" instead of dict.clear().

I don't understand underlying processes and why I had this issues.
In my opinion if you're leaving the function scope you don't have to 
worry about what variables you're leaving there. There are other guys at
 StackOverflow who has similar problems with big dictionaries and memory 
 but they're using different approaches.

Thanks you everyone for your time and advices! I think the problem 
is solved and I hope some one will find this helpful


More information about the Python-list mailing list