Question about asyncio and blocking operations

[Chris Angelico]

On Thu, Jan 28, 2016 at 8:13 PM, Frank Millman <frank at chagford.com> wrote:
> Run the database handler in a separate thread. Use a queue.Queue to send
> requests to the handler. Use an asyncio.Queue to send results back to the
> caller, which can call 'await q.get()'.
>
> I ran a quick test and it seems to work. What do you think?

My gut feeling is that any queue can block at either get or put. Half
of your operations are "correct", and the other half are "wrong". The
caller can "await q.get()", which is the "correct" way to handle the
blocking operation; the database thread can "jobqueue.get()" as a
blocking operation, which is also fine. But your queue-putting
operations are going to have to assume that they never block. Maybe
you can have the database put its results onto the asyncio.Queue
safely, but the requests going onto the queue.Queue could block
waiting for the database. Specifically, this will happen if database
queries come in faster than the database can handle them - quite
literally, your jobs will be "blocked on the database". What should
happen then?

The other risk is that the wrong result will be queried (two async
tasks put something onto the queue - which one gets the first
result?), which could either be coped with by simple sequencing (maybe
this happens automatically, although I'd prefer a
mathematically-provable result to "it seems to work"), or by wrapping
the whole thing up in a function/class.

Both of these are risks seen purely by looking at the idea
description, not at any sort of code. It's entirely possible I'm
mistaken about them. But there's only one way to find out :)

By the way, just out of interest... there's no way you can actually
switch out the database communication for something purely
socket-based, is there? PostgreSQL's protocol, for instance, is fairly
straight-forward, and you don't *have* to use libpq; Pike's inbuilt
pgsql module just opens a socket and says hello, and it looks like
py-postgresql [1] is the same thing for Python. Taking something like
that and making it asynchronous would be as straight-forward as
converting any other socket-based code. Could be an alternative to all
this weirdness.

ChrisA
[1] https://pypi.python.org/pypi/py-postgresql