[Python-Dev] Support for async read/write

[James Y Knight]

On Oct 19, 2010, at 1:47 PM, exarkun at twistedmatrix.com wrote:

> Adding more platform wrappers is always nice.  Keep in mind that the quality of most (all?) aio_* implementations is spotty at best, though. On Linux, they will sometimes block (for example, if you fail to align buffers properly, or open a file without O_DIRECT, or if there are too many other aio operations active on the system at the time, etc).  

You're thinking of the linux-specific AIO calls. Those have the properties you're describing (which makes them pretty useless for most code too), but they're completely different from the aio_* functions.

The POSIX aio_* calls don't do any of that. They aren't syscalls implemented in the kernel, they're implemented in glibc. They "simply" create a threadpool in your process to call the standard synchronous operations, and make it difficult to reliably get completion notification (completion notification takes place via Real-Time signals (SIGEV_SIGNAL), which can be dropped if linux runs out of space in its RT-signal-queue, and when that happens you get no indication that that has occurred. You can also do completion notification via calling a function on a thread (SIGEV_THREAD), but, for that, glibc will always spawns a brand new thread for each notification, which is quite slow.)

Basically: you shouldn't ever use those APIs. Especially on linux, but probably everywhere else. 

So, in conclusion, I disagree that adding wrappers for these would be nice. It wouldn't. It would cause some people to think they would be useful things to call, and they would always be wrong.

James