Question about asyncio and blocking operations

[Ian Kelly]

On Thu, Jan 28, 2016 at 9:40 AM, Frank Millman <frank at chagford.com> wrote:
> I have hit a snag. It feels like a bug in 'await q.get()', though I am sure
> it is just me misunderstanding how it works.
>
> I can post some working code if necessary, but here is a short description.
>
> Here is the database handler - 'request_queue' is a queue.Queue -
>
>        while not request_queue.empty():
>            return_queue, sql = request_queue.get()
>            cur.execute(sql)
>            for row in cur:
>                return_queue.put_nowait(row)
>             return_queue.put_nowait(None)
>            request_queue.task_done()

As I commented in my previous message, asyncio.Queue is not
thread-safe, so it's very important that the put calls here be done on
the event loop thread using event_loop.call_soon_threadsafe. This
could be the cause of the strange behavior you're seeing in getting
the results.

> The caller requests some data from the database like this.
>
>    return_queue = asyncio.Queue()
>    sql = 'SELECT ...'
>    request_queue.put((return_queue, sql))

Note that since this is a queue.Queue, the put call has the potential
to block your entire event loop.