memory consumption

[Alexey]

четверг, 1 апреля 2021 г. в 15:27:01 UTC+3, Chris Angelico:
> On Thu, Apr 1, 2021 at 10:56 PM Alexey <zen.su... at gmail.com> wrote: 
> > 
> > Found it. As I said before the problem was lurking in the cache. 
> > Few days ago I read about circular references and things like that and 
> > I thought to myself that it might be the case. To build the cache I was 
> > using lots of 'setdefault' methods chained together 
> > 
> > self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})... 
> > 
> > and instead of wring a long lines I decided to divide it to increase 
> > readability 
> > 
> > cluster = self.__cache.setdefault(cluster_name, {}) 
> > database = database.setdefault(database_name, {}) 
> > ... 
> > and I guess that was the problem. 
> > 
> > First thing I did was to rewrite this back to single line.
> If the cache is always and only used in this way, it might be cleaner 
> to use a defaultdict(dict) instead of the setdefault calls. Or, since 
> this appears to be a two-level cache: 
> 
> self.__cache = defaultdict(lambda: defaultdict(dict)) 
> 
> and then you can simply reference 
> self.__cache[cluster_name][database_name] to read or update the cache.

I agree

> Having that be more efficient than either self.__cache=None or del 
> self.__cache (which will be equivalent), I can understand. But better 
> than clearing the dict? Seems very odd. 

In this particular case 'cache.clear()' just don't work (in context of 
releasing memory). If someone can tell me why, I'll be very thankful 

> Ideally, though, you'd want to NOT have those reference loops. I 
> presume the database objects need to have a reference to whatever 
> 'self' is, but perhaps the cache can be done externally to the object, 
> which would make all the references one-way instead of circular. But 
> that's something only you can investigate. 

I did some refactoring and changed my code already.