[New-bugs-announce] [issue43119] asyncio.Queue.put never yields if the queue is unbounded

[Spencer Nelson]

New submission from Spencer Nelson <swnelson at uw.edu>:

I am writing some software that reads records from a very large file (~hundreds of GB), putting them in an `asyncio.Queue` as it goes, and a chain of consumers handle each record and do stuff over the network.

To my surprise, my program runs out of memory because the Queue producer coroutine never yields control. I think (but am not sure) that the asyncio.Queue.put method has no preemption point if the queue is not full; I was using an unbounded Queue, so it was _never_ full, so my coroutine was never unscheduled.

I have attached a file with a minimal reproducer. It creates an unbounded queue. A 'publish' task calls `queue.put` from an infinite sequence. A 'subscribe' task calls `queue.get`, and prints each item. Nothing gets printed if I run this, because the `queue.put` never blocks.

I would expect that `await queue.put` would occasionally cede execution to any other runnable coroutines, even if the queue is unbounded.

----------
components: asyncio
files: never_yields.py
messages: 386454
nosy: asvetlov, spenczar, yselivanov
priority: normal
severity: normal
status: open
title: asyncio.Queue.put never yields if the queue is unbounded
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file49789/never_yields.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43119>
_______________________________________