[issue43911] Queue.get() memory leak

[Jens]

Jens <multiks2200 at gmail.com> added the comment:

Hi,

Thanks for your reply, so I've run same script with queue.PriorityQueue, queue.LifoQueue, queue.SimpleQueue, Asyncio.Queue as well as collections.dequeue

1. PriorityQueue
>#########
>del_after_puts False del_after_gets True n_puts 20000000
>before run
>mem_pct 0.15% 
>------ put done  ----- qsize 20000000
>mem_pct 37.49% 
>------ gets done  ----- qsize 0
>mem_pct 0.16% 
>deleting queue after gets <queue.PriorityQueue object at 0x7f097cfeb290>
>mem_pct 0.16% 

2. LifoQueue
>#########
>del_after_puts False del_after_gets True n_puts 20000000
>before run
>mem_pct 0.15% 
>------ put done  ----- qsize 20000000
>mem_pct 37.49% 
>------ gets done  ----- qsize 0
>mem_pct 0.16% 
>deleting queue after gets <queue.LifoQueue object at 0x7fd1150c1250>
>mem_pct 0.16% 

3. SimpleQueue
>#########
>del_after_puts False del_after_gets True n_puts 20000000
>before run
>mem_pct 0.15% 
>------ put done  ----- qsize 20000000
>mem_pct 37.49% 
>------ gets done  ----- qsize 0
>mem_pct 0.28% 
>deleting queue after gets <_queue.SimpleQueue object at 0x7f1da93d03b0>
>mem_pct 0.28% 

4. asyncio.Queue
>#########
>del_after_puts False del_after_gets True n_puts 20000000
>before run
>mem_pct 0.23% 
>------ put done  ----- qsize 20000000
>mem_pct 37.69% 
>------ gets done  ----- qsize 0
>mem_pct 2.3% 
>deleting queue after gets <Queue maxsize=0>
>mem_pct 0.25% 

5. collections.deque
>#########
>del_after_puts False del_after_gets True n_puts 20000000
>before run
>mem_pct 0.15% 
>------ put done  ----- qsize 20000000
>mem_pct 37.61% 
>------ gets done  ----- qsize 0
>mem_pct 2.22% 
>deleting queue after gets deque([])
>mem_pct 0.16% 

So from the result it can be seen, that PriorityQueue and LifoQueue dont leak at all, and return to original memory after queues are emptied, so deleting the object reference has no effect.

SimpleQueue seems to leak in the same way as Queue, but to a way lesser extent and deleting object does not return the memory.

asyncio.Queue and collections.deque seem to leak to the same extent as Queue, but memory can be returned by deleting the object.

As result, this got be to just trying to use Queue._put() and Queue._get() to populate the deque inside the Queue object and bypass all conditions/locks logic, which produces the following result:
>#########
>del_after_puts False del_after_gets True n_puts 20000000
>before run
>mem_pct 0.15% 
>------ put done  ----- qsize 20000000
>mem_pct 37.61% 
>------ gets done  ----- qsize 0
>mem_pct 2.22% 
>deleting queue after gets <queue.Queue object at 0x7f6a08e152d0>
>mem_pct 0.16%

It can be seen that for a Queue where only _put() and _get() methods are used, the memory still leaks, but can be released by deleting the object reference.

Perhaps that indicates a possible leak in the Condition class? I run a Queue with threads on my production applications that run for weeks straight, and the Queue leaks, and memory profiler always shows a leak at waiters_to_notify = _deque(_islice(all_waiters, n)) in Condition.notify()
I dont want to make it more confusing by bringing it up here since it might be not the same kind of leak, because I dont simply populate up to 20million objects and then pop them out of the Queue in my application, but from the memory profiling results I see here this seems to be related.

Thanks

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43911>
_______________________________________