mmap caching

[Ross Ridge]

George Sakkis wrote:
> It's around 400MB.

On Windows you may not be able to map a file of this size into memory
because of virtual address space fragmentation.  A Win32 process has
only 2G of virtual address space, and DLLs tend to get scattered
through out that address space.

> As I said, I cannot reproduce the MemoryError
> locally since I have 1GB physical space but IIRC the user who reported
> it had less.

Virtual address space fragmentation isn't affected by the amount of
physical memory in your system.  A system with 64MB of RAM might be
able to map a 400MB file while system with 3G of RAM might not be able
to map it because of how DLLs got loaded in to the process.

> Actually I am less concerned about whether a MemoryError
> is raised or not in this case and more about the fact that even if
> there's no exception, the program may suffer from severe thrashing due
> to constant swapping.

Well, that's what you're asking for when you use mmap.  The same
mechanism that creates virtual memory using a swap file is used to
create a virtual memory mapping of your file.  When you read from the
mmap file pages from the file a swapped into memory and stay in memory
until they need to be swapped out to make room for something else.  If
you don't want this behaviour, don't use mmap.

                                    Ross Ridge