Python and the need for speed

[Marko Rauhamaa]

Paul Rubin <no.email at nospam.invalid>:
> I'd be interested to know how to open a disk file asynchronously in a
> single-threaded Python program under Linux. As far as I can tell there
> is no way to do it.

Traditionally, disk access in Linux has been considered nonblocking.
There is AIO, but that hasn't been used much. I have seen quite a bit of
activity on the Linux FS mailing list recently on AIO so maybe that will
change in the future.

I believe the lack of asynchronous disk I/O is related to the grand
Solaris idea which Linux adopted: all memory is on a disk and RAM is
merely a cache. You can turn disk access into memory access with mmap()
and you can turn memory access into disk access with /proc/mem.

If you access a file and the relevant bytes are already cached in RAM
you get them instantaneously; otherwise you'll have to wait for a "page
fault" to be processed. If you need to execute a CPU instruction and the
relevant address is already cached in RAM, the CPU can proceed without
delay; otherwise the CPU will have to wait for a page fault to be
processed.

As swapping is no longer considered normal on modern computers, the
memory-disk duality doesn't seem all that practical anymore. Rather,
you'd like to treat the disk analogously to network access and keep RAM
access separate.


Marko