threading

[Marko Rauhamaa]

"Frank Millman" <frank at chagford.com>:

> Does reading from disk count as blocking? Strictly speaking I would
> have thought 'yes'.

You have touched upon a very interesting topic there.

I can't speak for Windows, but linux doesn't really let you control the
blocking of disk access. In fact, linux doesn't really let you control
the blocking of RAM access, either. RAM and the disk are considered two
sides of the same coin. It is very difficult to guarantee that a process
has all of its memory "cached" in RAM short of not having a physical
disk mounted.

There is what's known as AIO, and it's supposedly supported in linux,
but I have never seen anybody use it, and I don't know how well tested
it is. Also, I don't know how well it integrates with regular asyncio.

On the other hand, you don't know if disk access ever blocks. Quite
often you will find that the whole active part of the file system is
kept in RAM by linux.

My rule of thumb, two processes per CPU, should alleviate disk blocking
issues. When that isn't enough, you may be forced to write a small file
server that translates disk access to socket/pipe access.

Sockets and pipes are different beasts because, unlike files, they are
allocated memory buffers in the kernel memory. Also, they are accessed
strictly sequentially while files can be sought back and forth.

I do think it would be a nice addition to linux if they added a, say,
AF_FILE socket type that provided a buffered socket abstraction for
physical files.


Marko