mmap caching

["Martin v. Löwis"]

George Sakkis schrieb:
>> You must be misinterpreting what you are seeing. It's the operating
>> system that decides what part of a memory-mapped file are held in
>> memory, and that is certainly not without limits.
> 
> Sure; what I meant was that that whatever the limit is, it's high
> enough that a MemoryError is raised before the limit is reached.

The operating system will absolutely, definitely, certainly release
any cached data it can purge before reporting it is out of memory.

So if you get a MemoryError, it is *not* because the operating system
has cached too much data.

In fact, memory that is read in because of mmap should *never* cause
a MemoryError. Python calls MapViewOfFile when mmap.mmap is invoked,
at which point the operating commits to providing that much address
space to the application, along with backing storage on disk
(typically, from the file being mapped, unless it is an anonymous
map). Later access to the mapped range cannot fail (except for
hardware errors), and if it would, you wouldn't see a MemoryError.

> It's the one in the 'Processes' tab of the Windows task manager (XP
> proffesional). By the way, I ran the same program on a box with more
> physical memory and the mem. usage stops growing at around 430MB, by
> which time the whole file is most likely cached. I'd be interested in
> any suggestions other than "buy more RAM" :) (these are not my machines
> anyway).

As a starting point, try understanding better what is really happening.
Turn on "Virtual Memory Size" in "View/Select Columns" also, and perhaps
a few additional counters as well. Also take a look at the "Commit
Charge", which takes into account swap file usage as well. Try
increasing the size of the swap file.

Regards,
Martin